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ABSTRACT 


A dramatic increase in the past three years in the 
Indonesian defense budget, from approximately US$800 
million (FY2002$) in 2002 to approximately US$ 2.4 billion 
(FY2005$) in 2005, has created the need for the Indonesian 
Department of Defense (DOD) and National Armed Forces (TNI) 
to enhance performance and accountability for effective and 
efficient use of state funding. It is imperative that DOD 
and TNI move to better inform the public and high level 
government officials by increasing transparency on how the 
defense budget is allocated and spent. Current DOD and TNI 
performance evaluation tools and financial management 
measurement and reporting methodologies are ineffective, 
particularly because they do not link performance 
assessment to the budget and the budget process. 

Reform is needed to implement Performance Based 
Budgeting (PBB) to create direct linkages between 
allocation of resources in the budget, performance 
measurement, and strategic planning relative to defense 
policy objectives so as to bolster public accountability 
for the DOD and TNI. This thesis analyzes implementation of 
the Government Performance Results and Act of 1993 in the 
US Federal Government and specifically in the US Department 
of Defense, to develop a PBB model for the Indonesian 
DOD/TNI. The thesis addresses critical factors and the 
steps and processes necessary to create a performance-based 
budget including strategic planning, performance 
measurement, and methods to link performance to budgets. 
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EXECUTIVE SUMMARY 


In essence. Performance Based Budgeting is a set of 
processes translating the strategic planning framework into 
performance measures in terms of input, output, and outcome 
to be tied into resource allocation in the budgeting 
process to seek greater efficiency and effectiveness. 

Past initiatives and experience as well as support 
from the President and Congress have made GPRA a prominent 
best practice to implement performance based budgeting in 
the US federal government. Specifically in the DOD, facing 
the complexities of its gigantic budget and business 
activities plus the daunting task inherent in developing 
measures of ultimate performance, progressively DOD has 
tried to incorporate links between budget development under 
0MB guidance and resource the allocation decision making 
process PPBS . 

To be able to implement PBB in 2006, the Indonesian 
Government and especially DOD has to better prepare the 
strategic framework and implementation guidance of PBB, 
especially the performance assessment tool to be used to 
link performance to budgets. There are two key challenges 
for the Indonesian DOD and TNI to succeed in implementing 
PBB. First, DOD and TNI must provide a unitary strategic 
plan that requires a total fusion of DOD and TNI goals and 
objectives. Second, PBB requires transparency. The degree of 
confidential justification in the defense sector needs to be 
examined and redefined so that transparency in cooperating 
with the stakeholders in designing performance measurement 
systems and linking performance to budgets will enhance 
public accountability in allocating and managing limited 
budget and resources. 
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I. 


INTRODUCTION 


A. BACKGROUND 

In the past two decades, there have been radical 
changes in public sector management. Although the issues 
between developed and developing countries are different, 
in essence, the main political theme is how to bolster 
public management accountability. Key stakeholders such as 
the public and the legislature have become increasingly 
interested in how public funds are allocated and spent and 
how to reduce public spending and to save the taxpayer 
money coupled with attempts to eliminate a large portion of 
moral hazard, the risk that the presence of action will 
affect other parties, and adverse selection, a situation 
where asymmetric information results in harming other 
parties, in the public sector. 

One of the attempts is to fundamentally restructure 
the budget management system. Shifting from the previous 
view of highly centralized budget management focused 
primarily on resource allocation and input control to a new 
budget management model, called "performance budgeting," 
aimed to create a direct linkage between allocating 
resources through the budget and performance integration in 
reaching stated objectives has become a prominent reform. 
Although the systems and implementations vary from country 
to country, the objective is solely directed to provide 
public sector budgeting information on an outcome basis 
seeking greater efficiency and effectiveness. 

The idea of performance budgeting in the United States 
was initiated in the 1950's based on Hoover Commission 
recommendation. However, changes in administration have 
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also created changes in the budget management system. From 
the Hoover recommendation, to Planning, Programming and 
Budgeting System (PPBS), Management by Objectives (MBO), 
Zero Based Budgeting, and finally Performance Based 
Budgeting has enriched the evolution of using performance 
measurement in budgeting process. 

The GPRA is the recent attempt of introducing 
performance-based management and budgeting techniques in 
the United States. The Recent economic downturn coupled 
with fiscal stress caused by September 11 attacks and its 
further effort in war of terrorism has created significant 
budget pressures not only in federal government but also 
state government as well as an increasing demand from the 
taxpayer for government efficiency and effectiveness. This 
emphasizes the need for reform. The information about the 
performance of programs has never been more important. 

The role of United States Department of Defense in 
performance budgeting evolution is very important. PPBS 
which was initiated by Robert McNamara, former Secretary of 
Defense, has provided an introduction of a decision-making 
framework to the executive branch budget formulation 
process by presenting and analyzing choices among long-term 
policy objectives and alternative ways of achieving them as 
well as a basis in developing performance plans and 
measurement system. 

Indonesia, as a developing country and fledging 
country in democracy, since a tragic monetary and economic 
crisis in 1998, has experienced a number of reform actions 
towards how government can run like a business. The country 
has faced remarkable challenges: economic reform, a 
transition to democracy after 30 years under military rule. 
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state owned enterprises restructuring and privatization, 
rising unemployment and poverty and a massive 
decentralization of power to nearly 400 local governments. 
The country still continues to suffer due to political and 
economic uncertainty. Other public accountability problems 
such as waste and inefficiencies, overseas loan management, 
and endemic corruption have really caught public attention 
and creditor interests. 

The effort to use performance budgeting in state 
government is still in process in setting up the 
performance measurement system and preparing the regulation 
to support the program. Although in some local governments 
performance budgeting has been initiated by having 
assistance from some Non Governmental Organizations (NGO), 
the standardized benchmarking performance measurement 
system is still not available. Past initiatives to 
implement a government performance evaluation tool had 
always failed because the government itself was not serious 
or even some feel reluctant to implement it. 

In the defense sector, a series of reform agenda has 
been undertaken to alter Indonesian National Armed Forces, 
TNI (Tentara Nasional Indonesia), and dominant role during 
the years of Soeharto's New Order regime into a new 
paradigm of military reform. Challenges such as controversy 
on division of authority between TNI and Department of 
Defense, military business under TNI Foundation funds, 
minimum defense budget allocation coupled with lack of 
deployment readiness due to inappropriate and obsolete main 
weaponry system, and inefficiency and ineffectiveness in 
resource allocation are still major concerns for the 
future. 
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DOD/TNI has already implemented the PPBS system in 
which initial efforts to measure performance had already 
begun. However, my organization. The Audit Board of the 
Republic of Indonesia faced difficulties in obtaining and 
evaluating information because there is no standard or 
benchmarking to be compared with and there is no law that 
forces them to disclose it to the Audit Board. 
Confidential justification to protect national security is 
always used to cover many issues in some areas of the 
military sector to avoid scrutiny from the stakeholders. 

As a result, the Audit Board, as an independent high 
state institution and government's accountability watchdog, 
will face enormous challenges in its function as an 
external control towards government accountability in 
utilizing and managing the state's finances to create good 
governance. The task is even much more difficult with the 
Law No. 7/2003 on State Budget because the law mandates the 
Audit Board to conduct audits on government performance. On 
the other hand, the current government performance 
evaluation tool still can not represent an ideal 
performance measurement system due to lack of strategic 
planning and inappropriate performance measures. 

Based on above circumstances, new tools to enhance 
government accountability and to cure all endemic and 
systematic government problems have to be found. Hopefully, 
Performance Based Budgeting will be one of them. 

B. OBJECTIVES 

The objective of this thesis is to investigate and 
analyze the implementation of Performance Based Budgeting 
(PBB) in US Federal Government especially in DOD as well as 
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to provide and promulgate a PBB model for Indonesian 
DOD/TNI. In addition, this research will address the 
historical perspective of PBB; steps and processes to 
implement PBB; the current implementation Government 
Performance and Results Act (GPRA) of 1993 including 
policy, guidance, tools and lessons learned; 
Recommendations will include why the PBB model is important 
for Indonesian DOD/TNI and how PBB might be implemented 
throughout the Indonesian DOD/TNI. To conclude, this study 
will provide conclusions and recommendations that could 
promulgate the implementation of PBB in all branches of the 
Indonesian government especially in DOD/TNI. 

C. RESEARCH QUESTIONS 

The research questions that will be investigated and 

analyzed in this project report are as following: 

• How has Performance Based Budgeting evolved 

historically in United States? 

• What are GPRA elements, budget and performance 

integration and Program Assessment Rating Tool? 

• How is the implementation of GPRA and specifically 
at Department of Defense? 

• How does the US experience differ from other 

countries (Britain, Canada, Australia)? 

• What is Performance Based Budgeting and what are the 
principles in PBB? 

• What are steps and processes that must be done in 
implementing Performance Based Budgeting? 

• How can performance measures be linked to budget? 
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• How is the current budgeting process in Indonesia 
and specifically at DOD/TNI? 

• How might Performance Based Budgeting be implemented 
trough Indonesian DOD? 

D. SCOPE AND LIMITATIONS 

This study provides a broad overview of performance 
based budgeting especially the Government Performance 
Results and Act (GPRA) 1993. The focus is on the strategic 
framework and the basic process of performance based 
budgeting to appropriately provide a basic model for PBB 
for Indonesian DOD/TNI. Due to the study solely based on 
the literature review, it does not address any detailed 
implementation of GPRA in the federal government and U.S. 
DOD as well as any detailed model including evaluation for 
Indonesian DOD/TNI. It simply provides the foundation for 
further action or research into the potential creation of 
such model as the performance based budgeting in Indonesia 
concept is still in the developmental stages. 

E. RESEARCH METHODOLOGY 

The methodological approach used in this MBA Project 
consists of conducting a literature review of Performance 
Based Budgeting theory, best practice, policy and 
guidelines, use of library information resources, and 
websites. It evolved as the research project and process 
for a period of approximately nine months. Initial stages 
were begun by performing an extensive literature review of 
the performance based budgeting in United States as well as 
the GPRA. It was followed by examining the conditions in 
Indonesia which includes the budgeting process and the 
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performance budgeting efforts. After deriving a number or 
lessons from those reviews, the focus shifted to a 
comprehensive review of performance measurements system by 
examining performance measurement theory, policy and 
guidelines, and best practice or worldwide benchmarking. 

F. ORGANIZATION OF STUDY 

The study is organized into five chapters. Chapter I 
provides the research background as well as delineates the 
objectives, research questions, scope and limitations and 
methodologies of the study. 

Chapter II describes PBB evolution in U.S., discuss 
GPRA including its elements, budget and performance 
integration, general implementation in federal government 
and specific implementation in DOD and finally provides 
some comparative studies in other countries, Britain, 
Canada, and Australia. 

Chapter III introduces the performance based budgeting 
and describes the literature review of Performance Based 
Budgeting including strategic planning, performance 
measures, and linking performance and budget. 

Chapter IV discusses the current condition in 
Indonesia including the budgeting process and performance 
budget efforts with focus on Indonesian Department of 
Defense and provides model and an example for Indonesian 
DOD about implementing the performance based budgeting. 

Chapter V provides the conclusions from the research 
concerning the performance based budgeting and 
recommendations for further study. 
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II. GOVERNMENT PERFORMANCE AND RESULTS ACT 1993 


A. INTRODUCTION 

This chapter lays out the literature review of the 
Government Performance and Results Act (GPRA) 1993. Section 
B describes the performance budgeting evolution in United 
States. Section C provides the GPRA overview and the 
followed by the GPRA main elements in Section D. Section E 
outlines the Budget and Performance Integration as well as 
the balance scorecard. Section F mainly discusses the 
Program Assessment Rating Tool (PART) system. Section G 
discusses general implementation of GPRA in Federal 
Government. Section H, reviews specifically the 

implementation of the GPRA in DOD. Section I reviews the 
comparative review of three countries, Britain, Canada and 
Australia. A summary is provided at the end of the chapter 
in Section J. 

B. PERFORMANCE BUDGETING EVOLUTION IN UNITED STATES 

Budgeting in the United States has experienced at 

least five emphases, starting with control at the turn of 

the century, moving to management in the New Deal and post- 

World War II period, to planning in the 1960s, 

prioritization in the 1970s and 1980s and to accountability 

in the 1990s. i GAO observes that: 

At the federal level, interest in performance 
budgeting has led to numerous initiatives since 
World War II, including four that were government 
wide in scope: (1) reforms flowing from the first 
Hoover Commission in its efforts to downsize the 


1 Charlie Tyer and Jennifer Willand, "Public budgeting in America: A 
twentieth century retrospective," Journal of Public Budgeting, 
Accounting & Financial Management 9, no. 2 (1997): 189. 
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post-World War II government, (2) Planning- 
Programming-Budgeting-System (PPBS) begun in 1965 
by President Johnson, (3) Management by 
Objectives (MBO) initiated in 1973 by President 
Nixon, and (4) Zero-Base Budgeting (ZBB) 
initiated in 1977 by President Carter. Each of 
these efforts established unique procedures for 
linking resources with results. 2 


Budget reform was initiated in 1910 by President Taft 


by forming the Commission on Economy and 
report from commission entitled The Need 
Budget was presented to Congress in 1912 
national attention on budgeting and 
management. Among major recommendations 
following: 


Efficiency. The 
for a National 
and it focused 
sound fiscal 
made were the 


1. The President should prepare and present a 
budget to Congress (the executive idea). 

2. A budget message should accompany the budget 
and should outline policy proposals of the 
President as well as include summary financial 
information. 

3. The Secretary of the Treasury should submit a 
consolidated financial report to Congress. 

4. Each Agency should submit to Congress an annual 
financial report. 

5. Agencies should establish and maintain a 
comprehensive accounting system.3 


These recommendations became the foundation for the 
Budget and accounting Act of 1921 which established an 
executive budget at the national level in the United States 
and became the foundation for present day budgeting at the 
federal level. 


2 United States. General Accounting Office, Performance budgeting : 
past initiatives offer insights for GPRA implementation, (Washington, 
D.C: The Office, 1997), 54. 

3 Cozzettom Kweit and Kweit, 1995: 20-21; in Tyer and Willand, Public 

budgeting in America: A twentieth century retrospective, 189. 
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Performance budgeting itself had been initiated since 
1950s based upon the 1949 Hoover Commission 

recommendations, first, as ".a focus of attention on the 

ends to be served by the government rather than on the 
dollars to be spent"4 and second, "...a budget based on 
functions, activities and projects, called a 'performance 
budget,' be adopted..." (Hoover Commission, 1949, p. 8.)- The 
Commission observed that if the federal budget were 
prepared on a performance basis, focusing attention on the 
amount of work to be achieved, and the cost of this work. 
Congressional action and executive direction of the scope 
and magnitude of different federal activities could then be 
appropriately emphasized and compared in the resource 
allocation process.5 

With the expansion of governmental activity during the 
New Deal and World War II, attention began to shift to 
management efficiency and led to wider interest in 
performance budgeting, particularly at the national level. 
Budget information for the national government would be 
structured in terms of activities rather than line items, 
and that performance measurement is provided along with 
performance report.6 

Consistent with the Hoover Commission's 

recommendations, the Congress enacted the Budget and 


4 Seckler-Hudson, C. Advanced Management.: 1953, reprinted in 1978. In 
Bernard Pitsvada and Felix LoStracco, "Performance budgeting—the next 
budgetary answer. But what is the question?" Journal of Public 
Budgeting, Accounting & Financial Management 14, no. 1 (2002) : 53. 

5 Jerry McCaffery and L. R. Jones, Budgeting and financial management 
in the federal government, (Greenwich, CT: Information Age Publishing, 
2001), 472. 

6 J. Burkhead, Government Budgeting, New York: John Wiley and Sons, 
1956, In Tyer and Willand, Public budgeting in America: A twentieth 
century retrospective, 189. 
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Accounting Procedures Act of 1950 (BAPA), which, among 
other things, required the President to present in his 
budget submission to the Congress, the "functions and 
activities" of the government, ultimately institutionalized 
as a new budget presentation: "obligations by activities."'^ 
GAO further states that these presentations were intended 
to describe the major programs, projects, or activities 
associated with each federal budget request—in a sense, the 
"performance budget" of a government which at that time was 
primarily involved in directly providing specific goods and 
services. Workload and unit cost information began to 
appear in the President's Budget, associated with the 
"obligations by activities" presentations, providing a 
means of publicly reporting the outputs of federal 
spending. 

Particularly at the national level, however, 
performance budgeting encountered a number of problems and 
performance budgeting did not work very well. Budget 
estimates were no more meaningful than those in line item 
budgets, works measurement presented problems, measuring 
the output of governmental services was imprecise, input 
could be easily measured but not outputs, and, finally, 
there was a lack of tools to deal with the long range 
problems.8 

When Robert McNamara led the Defense Department, he 
initiated a specific form of program budgeting called 
Program, Planning and Budgeting System (PPBS) in 1962 and 


' United States. General Accounting Office, Performance budgeting : 
past initiatives offer insights for GPRA implementation, 54. 

8 Miller, G.J., Productivity and the Budget Process," in Tyer and 
Willand, Public budgeting in America: A twentieth century 
retrospective, 189. 
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in few years. President Lyndon Johnson extended PPPBS to 


the entire federal government. Performance budgeting, as a 
subset of program budgeting, suffered the same fate as 
program budgeting in most agencies when the Nixon 
Administration abandoned it in 1969. However, where 
remnants of PPBS survived, so did performance measurements; 
in some respects, the Pentagon led the recent march back to 
performance budgeting because PPBS remains its operative 
budget system.9 

According to Allen Schick,lo several other arguments 
that made PBBS failed are: 


• It was introduced across the board without much 
preparation 

• It flew in the face of existing budgetary 
traditions and relationships; in particular, 
many people strongly objected to the suggestion 
that the budgetary process, which is inherently 
political, could be made "rational." 

• It was not given adequate resources, and top 
managers were not entirely committed to it. 

• Good analysis and data were in short supply, 
and they were necessary to produce the kind of 
information crucial to the success of PPBS. 

• Because PPBS was intended to require a review 
of all activities in each year, the reform 
caused so much conflict that the political 
system was not able to handle it. 

• PPBS clearly assumed that efficiency was the 

primary value to be considered in evaluating 
the usefulness of programs, which was not 
generally agreed to and is a difficult 

assumption to make in the case of a public 
program. 


" United States. General Accounting Office, Performance budgeting : 
past initiatives offer insights for GPRA implementation, 54. 

10 Schick, Allen. "A Death in the Bureaucracy: The Demise of Federal 
PPB." Public Administration Review, 1973, 33(2), 146-156. In Philip G. 
Joyce, "Performance Based Budgeting," in Handbook of Government 
Budgeting, ed. Roy T. Myers. Anonymous (San Fransisco: Josey-Bass 
Publishers, 1999), 597-619. 
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• PPBS was an executive budget system, and 
largely ignored the role of Congress in the 
budgetary process. The result was that Congress 
ignored the system in favor of its established 
procedures. 

According to GAO in its 1997 report, the performance 
measurements or performance factors became vital components 
in PPBS analyses and PPBS had successfully introduced a 
decision-making framework to the executive branch budget 
formulation process by presenting and analyzing choices 
among long-term policy objectives and alternative ways of 
achieving them. Performance was generally defined as agency 
outputs, with an agency's program structure linking outputs 
to long-term objectives. Systems analysis and other 
sophisticated analytical tools were an intrinsic part of 
PPBS, with measurement seen as an essential means to better 
understand federal outputs, benefits, and costs.n 

In 1973, President Nixon initiated Management by 
Objectives, which was primarily a federal management 
improvement initiative. GAO observes that MBO put in place 
a process to hold agency managers responsible for achieving 
agreed-upon outputs and outcomes. Agency heads were 
accountable to achieve presidential objectives of national 
importance; managers within an agency would be held 
accountable for objectives set jointly by supervisors and 
subordinates. Performance was primarily defined as agency 
outputs and processes, but efforts were also made to define 
performance as the results of federal spending—what would 
today be called "outcomes." 


United States. General Accounting Office, Performance 
past initiatives offer insights for GPRA implementation, 54. 
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Initiated in 1977 by President Carter 


Zero-Base 


Budgeting (ZBB), an executive branch budget formulation 

process, was introduced into the federal government. Zero 

Based Budgeting, a bottom-up concept in which all programs 

have no funding at the beginning of each budget cycle, 

required all departments to identify their programs, and to 

rank and justify the existence of each program in order of 

priority. GAO claims that: 

ZBB main focus was on optimizing accomplishments 
available at alternative budgetary levels and 
under ZBB agencies were expected to set 
priorities based on the program results that 
could be achieved at alternative spending levels, 
one of which was to be below current funding. In 
developing budget proposals, these alternatives 
were to be ranked against each other sequentially 
from the lowest level organizations up through 
the department and without reference to a past 
budgetary base. In concept, ZBB sought a clear 
and precise link between budgetary resources and 
program results. 12 


Related to the idea of Hoover Commission, PPBS, MBO 
and ZBB, according to GA0i3, past initiatives, although 
generally perceived as having fallen far short of stated 
goals, contributed to the evolution of performance-based 
measurement and budgeting in the federal government, and 
many concepts introduced by these initiatives became 
absorbed in the federal government and persisted long after 
their origins in PPBS, MBO, or ZBB had been forgotten. For 
example, first, the "obligations by activities" 
presentations established in response to the Hoover 


12 United States. General Accounting Office, Performance budgeting : 
past initiatives offer insights for GPRA implementation, 54. 

13 ibid. 
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Commission's performance budgeting recommendations continue 
today, although they are now referred to as "program 
activities." Second, PPBS and MBO fostered exploration of 
difficult performance measurement issues, ultimately 
demonstrating the inherent limitations of analysis in a 
political environment and the often complex and uncertain 
relationship between federal activities, outputs, and 
outcomes. Finally, ZBB illustrated the usefulness of 
presenting alternative funding levels and expanded 
participation of program managers in the budget process. 

When viewing collectively the past initiatives, GAO 
suggests two common themes. First, any effort to link plans 
and budgets must explicitly involve both branches of 
government, executive and Congress. The failure of PPBS and 
ZBB was solely caused by lack of Congressional involvement. 
Second, with all past initiatives and efforts from Hoover 
Commission, PPBS, and ZBB, it is more likely that the 
concept of performance continues to evolve, 

During the Reagan presidency, major adds to defense 
spending compared to cuts on non-defense spending, led to 
enormous increasing spending on discretionary accounts that 
caused incremental reductions to federal government agency 
programs and substantial growth in the federal budget. By 
the end of 1990s, the country condition was even worst with 
dollars overflowing the Treasury, the caps could not 
restrain political will and the budget continued to expand. 


United States. General Accounting Office, Performance 
past initiatives offer insights for GPRA implementation, 54. 
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All of this leads to think about implementing performance 
budgeting, is 

In the 1990s, considerable attention has been 
refocused upon performance budgeting, resulting in a new 
notion of performance budgeting to what called 
"Entrepreneurial Budgeting.16 One influence was the 1992 
publication of "Reinventing Government, How the 
Entrepreneurial Spirit is Transforming the Public Sector" 
by David Osborne and Ted Gaebler. The idea was to hold 
governments accountable for results rather than focus upon 
inputs as traditional budgets and management did. Cost 
saving and entrepreneurial spirit would be rewarded. A long 
term view would be facilitated in terms of strategy, costs 
and planning for programs.n 

Drawing on this attention, the Government Performance 
and Results Act (GPRA) of 1993 (PL. 103-62) was passed by 
Congress. The GPRA drew upon earlier antecedents, such as a 
report submitted during the Reagan Administration and a 
bill introduced in 1991 by Senator William Roth that had 
not been enacted, but was revised and became the GPRA. is 
Initially, the act specifically focused attention on 
results and performance budgeting. It requires federal 
agencies to prepare strategic plans by 1997, to prepare 
annual performance plans starting with fiscal year 1999, 


15 H. Heclo, Executive Budget Making. In Pitsvada and LoStracco, 
Performance budgeting—the next budgetary answer. But what is the 
question?, 53. 

16 D.A. Cothran, Entrepreneurial Budgeting: An Emerging Reform, (1993) 
in Miller, G.J., Productivity and the Budget Process," in Tyer and 
Willand, Public budgeting in America: A twentieth century 
retrospective, 189. 

17 ibid. 

18 w. Groszyk, Implementation of the Government Performance and Results 
Act of 1993, 1995, November, in ibid. 
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and submit an annual program performance report to the 
President and Congress comparing actual performance with 
their plans beginning in the year 2000. 

Basically, the new performance budgeting has been 
drawn from previous performance budgeting ideas and program 
budgeting/PPBS concept as well. The mechanism for the 
transformation of budgeting was a blend of rationality and 
Reaganism.i9 Also, like the previous reforms, performance 
budgeting was built on assumptions about the ability of a 
change in process to change outcomes, without understanding 
the mechanism by which that would be accomplished in a 
political system.20 

An important distinguishing aspect of GPRA is that it 
is a law; the others, PPBS and ZBB, were established 
through Presidential directive. Therefore, GPRA is created 
by both President and Congress and. It is expected that 
GPRA would not repeat the failure its predecessor. To 
maintain the efforts, 0MB needs to be more highly 
cooperative not only with agencies but also with other 
stakeholders, GAO and more importantly the Congress. 

C. GPRA OVERVIEW 

For decades, there has been a perception from the 
public that American government institutions and Congress 
are not working well and held in low esteem by the American 
People. The Committee on Government Affairs believes that 
part of the explanation for this apparent inconsistency 


I" Janet M. Kelly, "The long view: Lasting (and fleeting) reforms in 
public budgeting in the twentieth century," Journal of Public 
Budgeting, Accounting & Financial Management 15, no. 2 (2003) : 309. 

20 R.c. Moe, The "reinventing government" exercise: misinterpreting the 
problem, misjudging the consequences. Public Administration Review, 
54(2), 111-122, 1994. In ibid. 
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could be seen from a public opinion poll which showed that 
Americans, on average, believe that as much as 48 cents out 
of every Federal tax dollar is wasted. In other words, the 
public believes that it is not getting the level and 
quality of government service for which it is paying .21 

The committee also realizes that, first, waste and 
inefficiency in Federal programs undermine the confidence 
of the American people in the Government and reduce the 
Federal Government's ability to adequately address vital 
public needs. Second, Federal managers are seriously 
disadvantaged in their efforts to improve program 
efficiency and effectiveness, because of insufficient 
articulation of program goals and inadequate information on 
program performance. Finally, congressional policymaking, 
spending decisions and program oversight are seriously 
handicapped by insufficient attention to program 
performance and results. 

The purposes of GPRA are to-(l) improve the confidence 
of the American people in the capability of the Federal 
Government, by systematically holding Federal agencies 
accountable for achieving program results; (2) initiate 
program performance reform with a series of pilot projects 
in setting program goals, measuring program performance 
against those goals, and reporting publicly on their 
progress; (3) improve Federal program effectiveness and 
public accountability by promoting a new focus on results, 
service quality, and customer satisfaction; (4) help 
Federal managers improve service delivery, by requiring 
that they plan for meeting program objectives and by 


21 Committee on Governmental Affairs United States Senate, Government 
Performance and Resultas Act of 1993 Report,Anonymous , 1993), 1-55. 
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providing them with information about program results and 
service quality; (5) improve congressional decision-making 
by providing more objective information on achieving 
statutory objectives, and on the relative effectiveness and 
efficiency of Federal programs and spending, and (6) 
improve internal management of the Federal Government. GPRA 
is a potentially far reaching piece of legislation and it 
seeks to make government agencies more effective and 
accountable by enhancing their strategic planning, 
establishing measurable performance objectives, and 
eventually tying agency budgets to the accomplishment of 
those objectives22 

The concept draws on (indeed, transplants) concepts 
from private sector management models and establishes a 
system in which market-like discipline could be used to 
improve federal government management. 23 as noted in 
President Bush's 2003 budget submittal, the ultimate 
objective is to move the budget debate from "what will the 
federal government spend?" to "what will the federal 
government achieve?"24 The Bush administration has continued 
and expanded the efforts of the Clinton administration to 
integrate performance information in the federal budget 
process and to increase managerial flexibility. The 
performance information from the GPRA process is to be used 

22 Manuel R. Gomez, "Exposure Surveillance tools needed in agency GPRA 
plans," American Industrial Hygiene Association Journal 59, no. 6 
(1998) : 371. 

23 Beryl A. Radin, "The Government Performance and Results Act (GPRA): 

Hydra-Headed Monster or Flexible Management Tool?" Public 
administration review 58, no. 4 (1998): 307-316. 

24 Office of Management and Budget, Budget of the United States 
Government, Fiscal Year 2003 (Washington, DC, 2002) : 47-52. in R. M. 
McNab and F. Melese, "Implementing the GPRA: Examining the Prospects 
for Performance Budgeting in the Federal Government," Public Budgeting 
and Finance 23, no. 2 (2003) : 73-95. 
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to score agencies on their performance, allocate (and 
reallocate) funds among competing programs, and consolidate 
and terminate unnecessary programs. 25 

According to Wildavsky and Caiden 26 ^ there are three 
key challenges for the GPRA to be considered a success. 

First, the GPRA should assist in the creation of an 
institutional framework conducive to forming consensus on a 
unique set of objectives among conflicting stakeholders. 
Second, GPRA implementation must support the ultimate 
stated objective of linking resources to results-or to 
relate data on program performance to appropriation account 
structures-for the conjectured efficiency gains to be 

realized. Finally, the GPRA must overcome a traditional 
system of budgeting that, while often criticized and the 
focus of almost continuous reform efforts, has survived to 
this day. 

D. GPRA MAIN ELEMENTS 

The conceptual entirety of GPRA is fused from three 

basic elements: (1) strategic planning, (2) performance 
measurement (which covers goal setting, measurement, and 
reporting and (3) greater flexibility for managers in 

return for greater accountability for results.27 

A strategic plan of program activities covers at least 
a five year period and is updated at least every three 
years. The strategic plan contains a comprehensive mission 

25 Office of Management and Budget, Budget of the United States 
Government, Fiscal Year 2003 (Washington, DC, 2002) : 47-52. in ibid. 

26 Aaron Wildavsky and Caiden, The New Politics of the Budgetary 
Process (New York: Harper Collins, 1992) in ibid. 

22 Groszyk, Implementation of the Government Performance and Results 
Act of 1993, 
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statement for the agency, a specification of general goals 
and objectives (including outcome-related goals and 
objectives) as well as how they will be achieved, the 
relationship between the general goals and objectives and 
performance goals, identification of key factors that could 
affect the achievement of the goals and objectives, and a 
description of the program evaluations used in establishing 
or revising the goals and objectives. Each agency has to 
present the plan to Congress and others interested in or 
affected by the plan (the stakeholders). 

The strategic plan must contain a description of how 
the agency intends to achieve the general goals and should 
cover the overall approach that will be taken over the time 
period covered by the plan, including a schedule for 
significant actions and the needed resources. It should 
indicate how the goals of the annual performance plans will 
be used to measure progress in achieving the general goals 
of the strategic plan, and the underlying basis for any 
assumptions or projections. 28 

The annual performance plan includes not only an 
agency's annual performance plan but also the Federal 
Government performance plan for the overall budget. Whereas 
the agency plans are submitted to 0MB for use in preparing 
the overall government-wide performance plan, the Federal 
Government plan is to be submitted to Congress as part of 
the Budget of the United States. 

The Annual performance plans provide the direct 
linkage between an agency's longer-term goals defined in 


2o Committee on Governmental Affairs United States Senate, Government 
Performance and Resultas Act of 1993 Report, 1-55. 
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the strategic plan and the daily basis activities. The 
plans include performance goals that define the level of 
performance to be achieved by a program activity in 
quantifiable, and measurable form; indicate the operational 
process, skills, technology, and resources required; 
provide performance indicators to be used in measuring 
relevant outputs; establish a basis for comparing actual 
program results with the established performance goals; and 
specify the means to be used to verify and validate 
measured values.29 

Annual program performance reports contain two main 
parts: a report on the actual performance achieved compared 
to the performance goals expressed in the performance goals 
plan; and of the steps to be taken to achieve those goals 
that were not met. The reports provide comparison of the 
actual performance of each program activity to the 
performance goals in the Annual Performance Plan for a 
fiscal year. In this report, the agencies have to explain 
what goals have or have not been achieved, why certain 
performance goals have not been met in how to provide plans 
for achieving the goals in the future or to modify the goal 
if it is not appropriate. 

The agency's performance report must be submitted to 
the appropriate authorization and appropriations committees 
of the Congress and upon request, the Committee can ask 
more specific information on the actual performance for any 
performance indicator established in its annual performance 
plan. Agencies are required to begin reporting performance 
trends, on a phased-in basis, so that for FY 2002 and 


2“ Radin, The Government Performance and Results Act (GPRA): Hydra- 
Headed Monster or Flexible Management Tool?, 307-316. 
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thereafter, performance data will be shown for each of the 
most recent four years.30 The annual performance reports 
provide feedback to managers, policymakers, and the public 
about the accomplishment of the resources expended or the 
original goals. The reports must correlate between 
performance measurement information and program evaluation 
findings to provide a clear picture of the agency's 
performance and its efforts at improvement. 

In the managerial flexibility section, 0MB had to 
select at least five agencies to participate in pilots from 
among eligible GPRA performance planning and reporting 
agencies during fiscal years 1994 to 1996. Agency proposals 
were to identify the constraining requirement to be waived, 
quantify how relief would affect performance, and compare 
the anticipated performance improvements with current 
performance levels versus levels that could be expected 
without the waiver.3i Performance report shall also describe 
the use and assess the effectiveness of any waiver of 
administrative requirements and controls, and summarize the 
findings of those program evaluations completed during the 
year covered by the report. If the agency has prepared a 
classified or non-public annex to its annual performance 
plan, then those same items shall be covered in a 
classified or non-public annex to the performance report.32 

Any agency is not allowed to have authority to waive 
administrative requirements established in law. However, if 

30 Committee on Governmental Affairs United States Senate, Government 
Performance and Resultas Act of 1993 Report, 1-55. 

31 Jonathan D. Breul, "The Government Performance and Results Act—10 
years later," The Journal of Government Financial Management 52, no. 1 
(2003) : 58 . 

32 Committee on Governmental Affairs United States Senate, Government 
Performance and Resultas Act of 1993 Report, 1-55. 
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an agency has authority under a law other than this Act to 
waive a statutory requirement or control, it may do so and 
need satisfy only the requirements of that law. 33 
Requirements and controls that may be waived are those 
imposed across the government agency which has a central 
management role, and cover areas such as financial 
management, personnel, supply buildings etc.34 

E. BUDGET AND PERFORMANCE INTEGRATION 

Budget and Performance Integration (BPI) is one of the 
most important of the reforms on the President's Management 
Agenda. 0MB made this point explicitly when first 
announcing the management agenda by stating, "Improvements 
in the management of human capital, competitive sourcing, 
improved financial performance, and expanding electronic 
government will matter little if they are not linked to 
better results."35 According to 0MB, the reasons why budget 
and performance integration is very important in 
President's Management Agenda because scarce federal 
resources are seldom be allocated with convincing accounts 
of the results and agency performance measures are 
improperly integrated into agency budget submissions and 
the management and operations of agencies. 

To solve the problems and improve financial 
performance, 0MB suggested several reform actions to the 


33 Committee on Governmental Affairs United States Senate, Government 
Performance and Resultas Act of 1993 Report, 1-55. 

34 Groszyk, Implementation of the Government Performance and Results 
Act of 1993, 

35 "Budget and Performance Integration: The Reform 
Available from http://www.john-mercer.com/bpi_reform.htm 
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Administration such as integrating performance review with 
budget decisions, identifying high quality outcome measures 
which accurately monitor the performance of programs, 
integrating more completely information about costs and 
program performance in a single oversight process which 
include budgeting for the full cost of resources where they 
are used, making budget programs and activity lines more 
parallel with outputs, and where useful, improving 
alignment of budget accounts.36 

By, this initiative, 0MB expects some expected results 
to be achieved so that federal managers would have better 
program performance and better cost control and 
accountability. First, 0MB expects agencies to implement 
performance budgeting, in which each agency and program 
funding to be based on tangible results and to use 
comparisons between various programs to make such budget 
decisions. Starting in 2003, the President's Budget will 
shift budgetary resources among programs devoted to similar 
goals to emphasize those that are more effective and the 
Administration will set performance targets for selected 
programs along with funding levels.37 

Second, 0MB expects that by early 2004, agencies will 
fully understand and facilitate the Program Assessment 
Rating Tool (PART) to obtain an immediate impact on 0MB' s 
decisions on program funding levels. Finally, 0MB expects 
agencies to have the necessary financial management and 
accounting systems. The principles of managerial cost 

36 "Budget and Performance Integration: Expected Results," [cited 
2004], Available from http://www.john-mercer.com/bpi_results.htm 
33 ibid. 
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accounting, and especially Activity Based Costing and Cost 
Management (ABC/M), are key enablers for the agency to 
realize the full benefits of performance-based budgeting 
through improved day-to-day program management. 0MB 
mentions the following as being among the expected long¬ 
term benefits of this reform: 

o "Better performance..." 

o "Better control over resources used and 
accountability for results by program 
managers..." 

o "Better service..." 

o "Standard, integrated budgeting, performance, 
and accounting information systems at the 
program level that would provide timely 
feedback for management and could be uploaded 
and consolidated at the agency and government 
levels. This would facilitate the goals of the 
President's initiative to improve financial 
performance." 38 

To provide a consistent approach to rating programs 
across the Federal government, 0MB suggested to use an 
assessment took called The Program Assessment Rating Tool 
(PART). "The Program Assessment Rating Tool (PART) is a 
series of questions designed to provide. The PART is a 
diagnostic tool that relies on objective data to inform 
evidence-based judgments and to assess and evaluate 
programs across a wide range of issues related to 
performance ."39 As a central to the budget and performance 
integration initiative, the PART is meant to strengthen the 
process for assessing the effectiveness of programs by 
making that process more robust, transparent, and 


3o "Budget and Performance Integration: Expected Results," [cited 
2004], Available from http://www.john-mercer.com/bpi_results.htm 
39 "Program Assessment Rating Tool (PART) : Overview," [cited 2004] . 
Available from http://www.john-mercer.com/omb_part.htm 
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systematic. The PART covers seven categories of Federal 
Programs are: Direct Federal Programs, Competitive Grant 
Programs, Block/Formula Grant Programs, Regulatory Based 
Programs, Capital Assets and Service Acquisition Programs, 
Credit Programs, and Research and Development. 

According to 0MB in its fiscal year 2006 PART 
Instruction,40 the PART strengthens and reinforces 
performance measurement under GPRA by encouraging careful 
development of performance measures according to the 
outcome-oriented standards of the law and by requiring that 
agency goals be appropriately ambitious. Therefore, 
performance measures included in GPRA plans and reports and 
those developed or revised through the PART process must be 
consistent. 0MB further states that the PART also extends 
the usefulness of performance information by defining 
programs that comprise activities about which management 
and budget decisions are made. 

In its review of the first year of PART, by performing 
a simple regression analysis, GAO^i found that PART scores 
have a positive and statistically significant effect on 
recommended levels in the president's budget, although great 
deal of variance was left unexplained. To examine the effect 
of program size on the results, GAO divided all programs 
equally into three groups-small, medium, and large- based on 
their fiscal year 2003 funding estimate. Regression then was 
performed for discretionary programs in each group. The 


“^0 Office of Management and Budget, 0MB's Program Assessment Rating 
Tool: FY 2006 PART Instructions, 

WWW.whitehouse.gov/omb/budget/fy2006/part.htmled. 

41 United States. General Accounting Office, Performance budgeting 
[electronic resource] : observations on the use of OMB's program 
assessment rating tool for the fiscal year 2004 budget, (Washington, 
D.C: U.S. General Accounting Office, 2004) 
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results suggest that the statistically significant effect of 
overall scores on budget outcomes exists only for the 
smaller programs. The regression analysis also shows that 
PART scores have a positive and statistically significant 
effect on discretionary program funding. Regression results 
for mandatory programs showed-as expected-no relationship 
between PART scores and the level of funding in the 
President's Budget Proposal. Assessment ratings, however, 
can potentially affect the funding for discretionary 
programs either in the President's Budget proposal or in 
congressional deliberations on spending bills.42 

F. GPRA IMPLEMENTATION IN THE FEDERAL GOVERNMENT 

Although the act was passed in 1993, the deadline for 
implementation was 1997, the time when all executive 
agencies were required to submit a five-year strategic plan 
to 0MB and Congress. To start the implementation of the 
GPRA, the act mandated a series of pilot projects to 
provide experience and lessons for government agencies. The 
pilot projects for performance measurement were undertaken 
in 28 designations, 14 cabinet departments and 14 
independent agencies. Those pilot projects were finally 
started in fiscal 1998 after being delayed for at least a 
year by 0MB. 

During earlier implementation, there were massive 
critiques of the adequacy of the act to fulfill its goals 
and objective as well as the willingness of the federal 
agencies to seriously implement it. In a few months in 

42 United States. General Accounting Office, Performance budgeting 
[electronic resource] : observations on the use of 0MB's program 
assessment rating tool for the fiscal year 2004 budget, (Washington, 
D.C: U.S. General Accounting Office, 2004) 
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1998, the Washington press (which is rarely interested in 
management issues) had focused unusual intention to the 
implementation of GPRA, reporting various pronouncements 
from the GAO, 0MB and the Republican leadership in Congress 
about the relative success or failure of federal agencies 
to comply with its initial requirements related strategic 
planning. 43 

Beryl Radin44 also claims that the directions for the 
implementation of GPRA that did emerge from 0MB were 
variable. She said that in some instances, 0MB staff acted 
as a cheerleader but did not provide clear instructions to 
agencies on what was expected, while in other instances, 
0MB provided the sophisticated equivalent of a paint-by- 
numbers exercise. She continues by observing that there was 
very little attention to process measures and the 
relationship between process activities and output or 
outcome performance. 

GAO in its 1999 report45 observes that there were a 
variety and complexity of agency planning and budget 
structures. These necessarily resulted in a range of 
approaches to present and link the performance information. 
Based on their analysis, GAO concluded that some agencies 
were able to develop informative approaches to connect 
budgetary resource to results, while some were not. Most of 
agencies still kept their plans separated with their budget 
submissions and only 6 of the 35 plans had been fully 


Radin, The Government Performance and Results Act (GPRA): Hydra- 
Headed Monster or Flexible Management Tool?, 307-316. 

44 ibid. 

45 United States. General Accounting Office, Performance Budgeting: 
Initial Experiences Under The Results Act in Linking Plans with 
Budgets... 162173... U.S. GAO. .Anonymous (General Accounting Office 
April 12:, 1999) 
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integrated into the agency's budget justification. This can 
be seen in following table: 


Table 1: Agencies Status on Allocation of Program Activity 
Funding to Performance Goals 


Nonnumeric characteristics 

Allocated 
program activity 
funding to 
performance 
goals 

Did not allocate 
program activity 
funding to 
performance 
goals 

Total 


Number 

Percent 

Number 

Percent 

Number 







Concentration of spending 

Spending in multiple accounts 

10 

42 

14 

58 

24 

Spending concentrated in one account 

4 

36 

7 

64 

11 

Program activity structure 

Not common 

13 

39 

20 

61 

33 

Common 

1 

50 

1 

50 

2 

Budget structure used in the performance plan 

Proposed change in structure 

3 

100 

0 

0 

3 

Used fiscal year 1998 structure 

11 

34 

21 

66 

32 

Integration of performance plan 

Separate from budget justification 

9 

31 

20 

69 

29 

Fully integrated with budget justification 

5 

83 

1 

17 

6 

Program activities linked to 

Strategic goal 

3 

50 

3 

50 

6 

Strategic objective 

3 

43 

4 

57 

7 

Performance goal 

7 

50 

7 

50 

14 

Other 

1 

33 

2 

67 

3 

None 

0 

0 

5 

100 

5 

Plan Associated dollars with 

Strategic goal 

5 

71 

2 

29 

7 

Strategic objective 

3 

50 

3 

50 

6 

Set of Performance goals 

4 

100 

0 

0 

4 

Performance goal 

0 

0 

1 

100 

1 

Other 

1 

50 

1 

50 

2 

None 

1 

7 

14 

93 

15 

Relationship of program activities to 
performance goals 

Simple relationship 

8 

100 

0 

0 

8 

Complex relationship 

6 

27 

16 

73 

22 


Source: From GAO Report: GAO/AIMD/GGD-99-67, Appendix Il46 


On January 18, 2001, based on review on the results of 
some performance budgeting pilots, in its report, 0MB 
reported that the pilots demonstrated that assuring further 
performance measurement improvements and steadily expanding 
the scope and quality of performance measures was 
paramount, and that the existing statute provides 

46 United States. General Accounting Office, Performance Budgeting: 
Initial Experiences Under The Results Act in Linking Plans with 
Budgets... 162173... U.S. GAO. .Anonymous (General Accounting Office 
April 12:, 1999). 
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sufficient latitude for such improvement.47 in the same 
report, 0MB also concluded that the pilots raised several 
key challenges about performance budgeting such as: 

• In many instances, measuring the effects of 
marginal, annual budget changes on performance 
is not precise or meaningful. 

• While continuing to change from an almost total 
reliance on output measures to outcome 
measures, it will be much more difficult to 
associate specific resource levels with those 
outcomes, particularly over short periods of 
time. 

• Establishing clear linkages between funding and 
outcomes will vary by the nature of the program 
and the number of external factors. 

• Delays in the availability of performance data, 
sometimes caused by agencies reliance on 
nonfederal program partners for data 
collection, will continue to present 
synchronization problems during the budget 
formulation.48 

In maximizing the effort to support GPRA, in April 
2001, 0MB introduced a new approach, a sweeping 
reorganization to "better integrate the budget analysis, 
management review and policy development roles. 0MB also 
merged OMB's separate budget and management functions into 
five resource management offices (RMOs). The new offices 
were assigned responsibility for not only formulating and 
reviewing the budget request, but also assessing program 
effectiveness, conducting mid and long range policy and 


47 United States. General Accounting Office, Results-Oriented 

Government: GPRA Has Established a Solid Foundatuon for Achieving 

Greater Results, (Washington D.C.: U.S. General Accounting Office, 
2004 ), 270 . 

48 ibid. 
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program analysis, and implementing government wide 
management policy and program evaluation. 49 

Jonathan D. Breul, a fellow and member of the Board of 
Trustees of the National Academy Public Administration and 
former senior advisor to the deputy director for Management 
in 0MB, elaborated his review on GPRA in 2003 after its 10 
years of implementation. Of all GPRA features, he found 
that three features of GPRA have worked and four features 
have not worked. 

Three features that have worked are bipartisan 
statutory framework, strategic planning, and performance 
measurement. For bipartisan statutory framework, the GPRA 
has been supported by both Democrat and Republican, in 
which it was conceived by Republican senator, passed under 
a Democrat committee chair and signed by Democrat 
President. Even, President Bush has articulated the GPRA 
through its budget performance integration as a key of his 
Management Agenda. 

In implementing strategic planning, there have been 
dramatic changes in federal agencies. Before GPRA 
implemented, there was no single strategic plan in the 
federal government; nowadays it exists in every department 
and agency. However, according to Breul, the requirement to 
update the plan at least once every three years has proved 
to be awkward because it is out of synch with the four-year 
presidential cycle. 

For performance measurement, currently the federal 
agencies have improved in designing their performance 
information as useful baselines in assessing future program 


4“ Breul, The Government Performance and Results Act—10 years later, 
58 . 
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Breul observes that developing 


performance. However, 
credible information on program results remains a work in 
progress, as agencies struggle not only to define their 
contribution to outcomes- which in many cases are 

influenced only partially by federal intervention with 
funds, regulations, etc.- but also to assure the 

reliability of their performance data. 

The four features of GPRA that have not worked are 
managerial flexibility pilots, performance budgeting 
pilots, government wide performance plan and a paper 
exercise. For managerial flexibility, of 61 waiver 
proposals submitted by pilot project agencies, 0MB did not 
approve any of them, and even three quarters of them were 
simply not approved for statutory or other reasons. For 
the remaining proposals, according to Breul, 0MB or other 
central management agencies developed compromises by using 
authorities that were already available independent of 
GPRA. Breul further claims several factors contributed to 
the failure of the managerial flexibility pilots: first 
were changes in federal management practices and laws that 
occurred after GPRA was enacted; second is that GPRA was 
not the only way agencies could seek relief. In addition to 
other pre-existing authorities. Vice President Gore's 
National Performance Review had launched about 185 
reinvention labs to provide managers added operational 
authority, quite independent of GPRA.50 

However, the pilot process did provide some important 
lessons for government wide implementation. Breul continues 
by observing that: 0MB and the agencies learned that the 

50 Breul, The Government Performance and Results Act—10 years later, 
58 . 
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burdens and constraints that confronted managers were often 
self-imposed by the agency itself or its parent department. 
For example, the Internal Revenue Services sought a waiver 
from a requirement that envelopes be purchased through the 
General Services Administration-even though there was such 
no requirement by GSA.si 

For performance budgeting pilots, the problem existed 

because the agencies spent nearly all of their time and 

effort ensuring that strategic and annual plans would be 

useful documents and met the requirements of the statute 

and less devoting their time to the real performance 

budgeting implementation. This condition made 0MB delay the 

start of the pilot projects by one year. The government 

wide performance plans also failed because few agencies 

took any interest in the plans and even 0MB did not issue 

it in the fiscal years 2002 or 2003. 

Finally, relating to his statement to consider GPRA as 

a paper work exercise, Breul claims that: 

".there is a risk of GPRA becoming a hollow, 

paperwork exercise, requiring reporting that is 
unrelated to the real work of Congress and the 
agencies. If one were to stack up all the GPRA 
documents produced for Congress last year, the 
pile would measure over a yard high. Reporting is 
not timely for decision makers, and is too 
voluminous for users to find useful information 
(overly footnoted, etc.) appropriators and 
executive branch decision makers often must wade 
through reams of paper to find a few kernels of 
useful information."52 


51 Breul, The Government Performance and Results Act—10 years later, 
58 . 

52 ibid. 
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In its March 2004 reportss, gaO states that the act has 
laid a strong foundation for a more results-oriented 
federal government. However, GAO found several challenges 
that exist to effective implementation of GPRA. The 
challenges are inconsistencies of top leadership in showing 
commitment to achieving results, report mixed results in 
use of performance information, managers confrontation with 
a range of human capital challenges, persistent challenges 
in setting outcome-oriented goals, measuring performance, 
and collecting useful data, crosscutting issues in 
successful GPRA implementation and limited use of 
performance information by Congress. 

G. DOD IMPLEMENTATION OF GPRA 

Defense operations involves over $1 trillion in 
assets, budget authority of about $310 billion annually, 
and about 3 million military and civilian employees and 
these operations represents one of the largest management 
challenges within the federal government. 54 with these 
complexities, the development of appropriate measures is a 
daunting task, one described as taking years, being 
intrinsically difficult, and requiring a disciplined 
approach to research and development of needed measures. 55 
Philip G. Joyce observes that: 

National defense is a particularly difficult area 

for which to develop measures of ultimate 

53 United States. General Accounting Office, Results-Oriented 
Government: GPRA Has Established a Solid Foundatuon for Achieving 
Greater Results, 270. 

54 United States. General Accounting Office, Status of Achieving 
Outcomes and Addressing Major Management Challenges, (Washington D.C.: 
U.S. General Accounting Office, 2001), 36. 

55 Larry C. Juul, DOD Implementation of the Government Performance and 
Results Act of 1553,Anonymous (Virginia: Defense Logistic Agency, 1997) 
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performance, perhaps because it is almost 
impossible to determine, without some military 
conflict, whether the objective of national 
defense is being achieved. Measurement is further 
complicated by the concept, as some would argue 
that the purpose of defense is not winning wars 
but preventing them. For this reason, many 
measures of performance for national defense 
often focus on the quality and readiness of 
military forces, assuming that a well qualified, 
appropriately trained military can best achieve 
whatever goals are paramount at the moment.56 

However, the efforts to align performance measures 
into budget would be much easier because DOD has already 
implemented the PPBS which provides a decision-making 
framework of budget formulation process and a basis of 
developing performance measurement system. 

Since the earliest implementation of GPRA, The 
Department of Defense has tried to incorporate the budget 
development in the context of 0MB guidance and in the 
context of its own budget development and resource 
allocation decision making process in PPBS. Explicitly, the 
Under Secretary of Defense (Comptroller) , who is also the 
person responsible for overseeing the implementation of 
GPRA within DOD, clearly outlined in a 1995 memorandum that 
general guidelines for DOD implementation include that the 
plans reflect the most important aspects of Defense; have 
DOD consensus; are integrated with the DOD Planning, 
Programming, and Budgeting System (PPBS), and are supported 
by functional managers who will be responsible for 
achieving the goals.57 


56 Joyce, 1993 In ibid. 

57 Office of the Under Secretary of Defense, Comptroller, 1995. 
Department of Defense Corporate Level Performance Goals and Measures 
under the GPRA - ACTION MEMORANDUM, Washington, DC (April 24) In ibid. 
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Initially, the DOD implemented the GPRA in two ways by 
sponsoring performance measurement pilot activities and by 
developing and securing internal consensus on DOD-wide 
corporate goals. Beginning in 1994, DOD also initiated to 
develop a set of DOD-wide corporate goals as part of GPRA 
implementation. Several DOD activities became the pilot 
projects: Defense Logistics Agency, Army Audit Agency, Army 
Corps of Engineers (Civil Works), Defense Commissary 
Agency, Air Combat Command, Navy Carrier Battle Group 
(CINCLANTFLT), and Army Research Laboratory. 

In 1995 DOD made several improvements toward the 
effort implementing the GPRA such as providing draft DOD 
mission and vision statements, and seven corporate goals 
were provided to the Joint Chiefs of Staff (JCS) and 

Under/Assistant Secretaries of Defense and issuing a new 
unclassified Annex A to the Defense Planning Guidance 
(DPG) , one of major documents in the PPBS cycle and a 

classified set of assumptions and instructions developed by 
the Office of the Assistant Secretary of Defense (Strategy 
and Requirements) and approved by the Deputy Secretary of 
Defense that the Components follow in preparing their 
Program Objectives Memorandum (POM) and budget. The most 

important idea in the unclassified DPG annex was the 
development of draft performance measures for use in the 

DOD DPRA performance plan. 

The instructions to the Military Services for 

preparation of the Department's FY 1997-2001 Program 

Objectives Memorandum (POM) issued in February of 1995 

included the following requirement "Components should 
review the DOD corporate level goals and develop 
performance measures that relate to them. During the 
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program review, components will provide a report to the 
Program Review Group that identifies these performance 
measures." 58 

In April 1996, GPRA implementation was further 
escalated by incorporating a GPRA annex in DPG and 
reflecting DOD mission, vision, and the six corporate level 
goals refined in the instructions for the FY 1998-2003 POM. 
These will be adjunct to the National Military Strategy and 
Quadrennial Defense Review which are intended to be used to 
satisfy the strategic planning requirements of the GPRA. 
Specific DOD mission, vision and corporate level goals have 
been developed and a current published rendition appears 
below: 59 


Table 2: United States DOD Mission, Vision, Goals 


DOD Mission 

Statement 

The mission of the Department of Defense is to support and 
defend the Constitution of the United States, to provide 
for the common defense of the United States, its citizens 
and its allies, and to protect and advance U.S. interest 
around the world 

DOD Vision 

Statement 

The Department of Defense: 

• Successfully defends the United States, its citizens, 
interests and allies 

• Fields the best trained, best equipped, best prepared 
joint fighting force in the world 

•Supports alliance/security relationships that 

protect/advance U.S. security interests 

•Advances national priorities in concert with Congress, 
other agencies and the private sector 

• Serves as a model of effective, efficient and innovative 
management and leadership 

DOD Corporate 

Level Goals 

1. Provide a flexible, ready, and sustainable military 
force capable of conducting joint operations to execute 
the National Military Strategy, including the capability 
to: 

• Rapidly project power to deter and, if necessary, 
fight and win two nearly simultaneous major regional 
conflicts (MRCs) in concert with regional allies. 

• Support friends and allies and shape the 

international environment in ways favorable to U.S. 
interests through peacetime overseas presence 


58 Office of the Secretary of Defense, Program Analysis and Evaluation, 
1995. FY 1997-2001 POM Preparation Instructions. Washington, DC. ibid. 

59 Maroni, Alice C., 1996. "DOD Implementation of the Government 
Performance and Results Act," Armed Forces Comptroller, Vol. 4, No.4, 
(Fall) In ibid. 
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• Conduct a wide range of contingency operations, 
including peace operations, counter-drug and counter¬ 
terrorist activities, noncombatant evacuations, and 
disaster assistance. 

• Deter, prevent and defend against the effective use 
of nuclear, biological, and chemical weapons and their 
delivery means and prevent the acquisitions of such 
weapons. 

2. Recruit and retain well-qualified military and civilian 
personnel and provide them with equal opportunity and a 
high quality of life. 

3. Sustain and adapt security alliances, enhance coalition 
war fighting, and forge military relationships that 
protect and advance U.S. security interests. 

4. Maintain U.S. qualitative superiority in key war 
fighting capabilities (e.g. information warfare, 
logistics). 

5. Reduce costs and eliminate unnecessary expenditures, 
while maintaining required military capability across 
all DOD mission areas by employing all modern management 
tools and working closely and effectively with other 
government agencies. Congress, and the private sector. 


By launching the Defense Reform Initiative report in 
1997, a reform plan for business operations, DOD 
established a management oversight structure to help 
sustain the direction and emphasis of these efforts. The 
requirement to develop annual performance contracts was 
started in fiscal year 1999. The Defense Reform Initiative 
was such a performance contracts which were internal 
management agreements between the principal staff 
assistants—also known as Under and Assistant Secretaries—in 
the Office of the Secretary of Defense, the Deputy 
Secretary, and the defense agencies. 

By this performance contract, DOD agencies were 
required to improve goals and objectives in terms of cost, 
productivity, quality, and responsiveness to customers and 
to assess the progress of achieving performance contract 
goals in annual performance reports. DOD intended the 
performance contracts to improve oversight of the defense 
agencies by (1) identifying programming and budgeting 
issues, (2) providing agency directors with clear 
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objectives and performance targets, (3) establishing open 
and direct communications with customers to, among other 
things, demonstrate how their actions affect agency 
performance, and (4) changing the way DOD does business.60 

To support the effort, two agencies were formed. First 
was the Defense Management Council to oversee Defense 
Reform Initiative efforts—chaired by the Deputy Secretary 
and consisting of key civilian and military leaders. Second 
was the Defense Agency Task Force to oversee the 
development and review of performance contracts—chaired by 
the Deputy Director of Resource Analysis in the Office of 
the Secretary of Defense, Program Analysis and Evaluation 
(OSD/PA&E) and consisting of senior-level representatives 
from the Under Secretaries of Defense, the service 
secretaries, and the Joint Staff. 

Eour defense agencies: the Defense Logistics Agency, 
the Defense finance and Accounting Service, the Defense 
Contract Audit Agency, and the Defense Health Program were 
selected to implement performance contracts during fiscal 
year 1999, and gradually expanded the requirement to a 
total of 10 agencies by fiscal year 2002. In 2001 
Quadrennial Defense Review Report, DOD uses the risk 
management framework to consider trade-offs among defense 
objectives and resource constraints. The framework consists 
of four dimensions of risk: 

• Force management— the ability to recruit, 

retain, train, and equip sufficient numbers of 
quality personnel and sustain the readiness of 


60 United States. Government Accountability Office, Defense Management: 
Tools for Measuring and Managing Defense Agency Performance Could be 
Strengthened, (Washington D.C.: U.S. Government Accountability Office, 
2004), 53. 
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accomplishing 


its many 


the force while 
operational tasks; 

• Operational— the ability to achieve military 
objectives in a near-term conflict or other 
contingency; 

• Future challenges— the ability to invest in new 
capabilities and develop new operational 
concepts to dissuade or defeat mid- to long¬ 
term military challenges; and 

• Institutional— the ability to develop management 
practices and controls that use resources 
efficiently and promote the effective operation 
of the defense establishment.6i 

To develop performance goals and report, DOD also gave 
the defense agencies the option to use a balanced 
scorecard. According to GAO, DOD requires 10 of 27 defense 
agencies with businesslike missions to have either 
performance plans or balanced scorecards, five defense 
agencies use performance plans, and the remainder have 
exercised the option to develop balanced scorecards. In 
addition, three intelligence agencies have developed 
performance plans in order to demonstrate how they are 
coordinating with DOD on strategic and budgetary planning 
in the post-September 2001 environment. 

Despite the above effort, GAO still feels 
unsatisfactory with DOD efforts. Based on its 2001 report62, 
GAO claims that DOD's progress in achieving the selected 
outcomes is unclear, because most of the selected program 
outcomes are complex and interrelated and may require a 
number of years to accomplish, and also because DOD did not 


61 United States. Government Accountability Office, Defense Management: 
Tools for Measuring and Managing Defense Agency Performance Could be 
Strengthened, (Washington D.C.: U.S. Government Accountability Office, 
2004), 53. 

62 United States. General Accounting Office, Status of Achieving 
Outcomes and Addressing Major Management Challenges, 36. 
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provide a full assessment of its performance. GAO continues 
by identifying some weaknesses in DOD strategies for 
achieving unmet performance goals in the future:63 


Table 3: Weaknesses in DOD strategies for achieving unmet 
performance goals in the future 


Planned Outcome 

Weaknesses 

The U.S. maintain 
technological 
superiority in key war¬ 
fighting capabilities 

• The extent to which DOD has made progress is 
unclear 

• Some of the performance goals underlying measures- 
such as procurement spending and defense technology 
objectives- do not provide a direct link toward 
meeting the goal and make it difficult to assess 
progress 

• The performance report does not reflect concerns 
raised about the adequacy of its strategy and the 
timely introduction of new technologies to 
operational forces. 

U.S. military forces 
are adequate in number, 
well qualified, and 
highly motivated 

• Performance measures do not adequately indicate its 
progress toward achieving the outcome of ensuring 
that U.S. military forces are adequate in number, 
well qualified, and highly motivated. 

• With the exception of the enlisted recruiting area, 
DOD does not identify specific planned actions that 
it or the services will take to assist them in 
meeting unmet performance targets in the future. 

• The report did not discuss the difficulties of the 
current retention environment, but the report 
contains little clear articulation of specific 
actions or strategies being taken to improve future 
retention 

Combat readiness is 
maintained at desired 
levels 

• The level of progress maintaining combat readiness 
at desired levels is unclear 

• Some performance measure targets had been lowered, 
were incomplete, or were not met which make 
difficult to assess the progress. 

Infrastructure and 
operating procedures 
are more efficient and 
cost effective 

• The extent of the progress to be more efficient and 
cost-effective is unclear 

• The targets such as disposing of property, reducing 
of its logistics response time, and streamlining 
the acquisition workforce do not hold up to 
scrutiny and some targets had been lowered or were 
not met. 

Reduced availability 
and/or use of illegal 
drugs 

•DOD FY2000 performance report did not include 
performance goals or measures related to its 
outcome, therefore no basis to assess the progress. 

Fewer erroneous 
payments to contractors 

• DOD had no performance goals directly related to 
achieving the outcome. 


63 United States. General Accounting Office, Status 
Outcomes and Addressing Major Management Challenges, 36. 
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of Achieving 





In February and August 2003, DOD issued guidance to 
refine its approach to performance management and renamed 
the contracts as performance plans. First implemented in 
September 2003, according to GAO, the performance plans 
build on the performance goals and measures established in 
the performance contracts and are intended to include a 
more comprehensive set of performance measures that align 
with agency strategic plans and department-wide goals set 
forth in DOD's risk management framework. 

In 2004 report,64 based on its review on three defense 
agencies DLA, DISA, and DODEA, GAO acknowledges that the 
performance contracts or performance plans were an 
important step toward more results-oriented management 
practices for the defense agencies, however, the contracts 
would have been more effective tools for informing decision 
making and managing and measuring agency performance had 
the agencies included additional attributes associated with 
results-oriented management. GAO further observes that 
contracts showed progress made and encouraged discussion 
among top agency and OSD leaders about actions needed to 
achieve targeted results, particularly in terms of quality, 
quantity, cost, and timeliness of services, however, DOD 
guidance did not require performance contracts to include 
certain key attributes to be associated with results- 
oriented management. 

The Mercatus Center, George Mason University, which 
has developed a seven-step process, called "Outcome-Based 
Scrutiny," that provides a framework for comparing the 


64 United States. Government Accountability Office, Defense 
Management: Tools for Measuring and Managing Defense Agency Performance 
Could be Strengthened, 53. 
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results and costs of programs with similar objectives and 
assessing the likely impact of reallocating resources to 
the most effective programs had assessed the agencies 
annual performance report scorecard. The purpose of the 
assessment is not to ascertain how well the agencies' 
reports inform the public about the results they produced 
so that policymakers and citizens may make informed 
judgments about the agencies' results. The evaluation 
factors are grouped under three general categories; does 
the agency report its accomplishments in a transparent 
fashion; does the report focus on documenting tangible 
public benefits the agency produced; does the report show 
evidence of forward-looking leadership that uses 
performance information to devise strategies for 
improvement ? 

In general, based on its assessment on FY 2003 and 
2004, DOD ranked as one of the lowest agencies which inform 
the performance to the public. The scoring comparison of 
DOD to other federal agencies for FY 2003 and FY 2004 can 
be seen in following table: 
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Table 4: Scoring Comparison Between DOD and Other Federal 
Agencies for FY 2003 and FY 2004 


Government Departments/Agencies 

FY 2003 Scoring^^ 

FY 2004 Scoring^^ 

T 

PB 

L 

Rank 

T 

PB 

L 

Rank 

Department of Labor 

18 

16 

14 

1 

17 

17 

16 

1 

Department of State 

15 

14 

14 

4 

18 

17 

14 

2 

Department of Transportation 

16 

15 

17 

1 

17 

16 

13 

3 

Department of Veterans Affairs 

17 

15 

15 

3 

17 

15 

14 

3 

Department of Commerce 

10 

11 

9 

16 

14 

11 

15 

5 

Small Business Administration 

10 

15 

11 

7 

13 

13 

13 

6 

Department of Justice 

10 

10 

11 

15 

14 

12 

13 

6 

Department of Energy 

13 

9 

10 

12 

14 

10 

15 

6 

United States Agency for International 
Development 

8 

9 

7 

21 

11 

14 

12 

9 

Environmental Protection Agency 

13 

11 

12 

7 

14 

12 

10 

10 

Nuclear Regulatory Commission 

13 

13 

10 

7 

13 

12 

11 

10 

National Science Foundation 

10 

9 

9 

17 

13 

12 

11 

10 

Department of the Interior 

13 

15 

12 

6 

11 

12 

12 

13 

Department of Agriculture 

15 

13 

15 

4 

14 

10 

11 

13 

Department of Education 

11 

10 

11 

12 

13 

10 

11 

15 

Department of Housing and Urban 
Development 

15 

9 

11 

10 

10 

11 

11 

16 

Department of the Treasury 

10 

8 

9 

19 

10 

12 

10 

16 

General Services Administration 

14 

11 

10 

10 

12 

9 

11 

16 

National Aeronautics and Space 

Administration 

9 

7 

9 

20 

10 

8 

14 

16 

Social Security Administration 

13 

8 

11 

12 

12 

8 

12 

16 

Office of Personnel Management 

9 

8 

11 

17 

9 

8 

13 

21 

Department of Homeland Security 

NA 

NA 

NA 

NA 

7 

9 

11 

22 

Department of Defense 

11 

6 

6 

22 

7 

6 

9 

23 

Department of Health and Human Services 

8 

7 

8 

22 

NA 

NA 

NA 

NA 


Specifically, for FY 2003, the Mercatus Center 
observes that for transparency, although the report could 
be located on DOD website and the performance information 
is confined to 30 pages in a 400 page document. Management 
Discussion and Analysis section lists 40 systemic and 
material weaknesses in the performance data and an appendix 
contains baseline and trend data back to fiscal 1999 or 
2000 for each goal. 

For public benefits, first, three of four strategic 
goals, and about half of the 13 performance goals are 


65 Mercatus Center. George Mason University, 5th Annual Performance 
Report Scorecard: Which Federal Agencies Best Inform the Public (FY 
2003J,Anonymous George Mason University. Mercatus Center, 2004) 

66 Mercatus Center. George Mason University, 6th Annual Performance 
Report Scorecard: Which Federal Agencies Best Inform the 
Public,Anonymous George Mason University. Mercatus Center, 2005) 
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outcomes; second, performance measures do not begin to 
capture the breadth and depth of Defense's operations and 
do not measure key results related to strategic goals; 
third, there is no discussion of the recent operations in 
Iraq and Afghanistan; and fourth, lack of information about 
the department's actual performance; and finally, 
voluminous financial information is not linked to strategic 
goals or measures. 

For leadership, first, performance shortfalls 
disclosed with minimal explanation; second, senior 
management offers no direct response to nine major 
management challenges identified by the inspector general, 
except some proposed legislative changes, third, six 
Defense issues remain on General Accounting Office's "High 
Risk List."; and fourth, the efforts are underway to 
transform the department, but offers few specifics. 

For 2004, for transparency, the Mercatus Center 
observes that first, the report is available via an 
unintuitive link under "DOD Websites." downloadable as a 
single PDF document or separate file; second, confusing mix 
of risk-management strategic objectives, "policy goals," 
and "performance metrics."; third, report omits legally 
required assessment by agency head of data shortcomings and 
efforts to address them, although portions of the report 
make clear that the department faces severe data problems; 
fourth, better explanation is needed for baseline and 
trend data back to fiscal 2001. 

For public benefits, first, four strategic objectives 
are outcome-oriented, and goals and measures are a mix of 
outcomes and activities; second, department states that the 
Quadrennial Defense Review Report serves as its strategic 
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plan; third, existing measures do not come close to 
capturing the full range of the department's operations; 
fourth, the department is in the process of developing many 
new performance measures; and finally, no information links 
goals and results to costs. 

For leadership, first, report says little about the 
department's most obvious accomplishments; second, when 
performance shortfalls are acknowledged, accompanying 
explanations are often perfunctory, third. Inspector 
general says the department has made progress on major 
management challenges but provides few specifics; fourth, 
some initiatives to improve on specific objectives are 
described; and finally. Deputy secretary's letter says 11 
management control weaknesses were corrected, but 46 
remain. 

H. COMPARATIVE EXPERIENCE WITH OTHER COUNTRIES 

1. Britain 

The performance initiative was started by launching 
the Financial Management Initiatives in 1982 which aims to 
quantify wherever feasible the assessment of whether 
programs are providing "value for money." It was introduced 
when Britain experienced a severe pressure on 
administrative expenditure and contraction in public 
employment. In announcing FMI, the government sought "in 
each department an organization and a system in which 
managers at all levels have a clear view of their 
objectives and the means to assess, and wherever possible. 
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measure outputs or performance in relation to those 


objectives. 67 

McCaffery and Jones observe that: 

The Financial Management Initiative called for a 
radical change in the internal structure and 
operations of government agencies. Objectives 
were to be assigned to responsibility centers. 
Costs were to be systematically identified. They 
were to be measured on and accrual bass (i.e. 
matching resources consumed to services 
delivered) and include not only the direct costs 
of service delivery but overheads as well. This 
identification enabled those responsible for 
meeting particular objectives to be held 
accountable for the cost or the resources they 

were consuming.68 


FMI does not require a uniform approach for all 
departments. Although there is a common element, an 
insistence on delegated budgeting in which responsibility 
for resources is pushed down the line to budget holders 
(those who actually spend resources and carry out 
operations) who are to be given sufficient flexibility and 
incentive to produce value for moneySS, each department is 
freely developing the managerial style and system suited to 
its circumstances as long as it is in accordance with 
certain broad guidelines. 

FMI conceives of budgeting as a "contract for 
performance" in which departments commit themselves to 
concrete targets in exchange for agreed resources. A 1986 


67 Cmnd. 8616, op.cit. in Allen Schick, "Budgeting for Results: Recent 
Developments in Five Industrialized Countries," Public administration 
review 50, no. 1 (1990) : 26-34. 

68 McCaffery and Jones, Budgeting and financial management in the 
federal government, 472. 

69 hM Treasury, Multi-departmental Review of Budgeting: Final Central 
Report, 1986, p. 8 in Schick, Budgeting for Results: Recent 
Developments in Five Industrialized Countries, 26-34. 
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Treasury report defined budgeting as a means of delivering 
value for money against a background of aims, objectives 
and targets and it will only fully realize its full 
potential if it has the support and involvement of 
management, if there are strong connections between 
budgets, outputs and results, and if it operates within a 
supportive central and managerial environment. 

The latest major development of central government 
performance measurement is Public Service Agreements (PSA), 
which was launched in 1998 and represents a fundamental 
change in the accountability of government to Parliament 
and the public. Each department (ministry) is required to 
provide its own PSA. Each PSA sets out the key performance 
targets which will contribute to achieving aims and 
objectives and includes an introduction which explains how 
these contribute to the Government's overall strategy to 
meet its key Comprehensive Spending Review (CSR) objectives 
and to improve productivity. 

PSA includes in single document important information 
on aims and objectives, resources, performance and 
efficiency targets and related policy initiatives. It has 
five key elements: 

• An introduction, setting out the Minister or 
Ministers accountable for delivering the 
commitments, together with the coverage of the 
PSA, as some cover other departments and 
agencies for which the relevant Secretary of 
State is accountable; 

• the aims and objectives of the department or 
cross-cutting area; 

• the resources which have been allocated to it 
in the CSR; 


70 hM Treasury, Multi-departmental Review of Budgeting: Executive 
Summary, 1986, in ibid. 
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• key performance targets for the delivery of its 
services, together with, in some cases, a list 
of key policy initiatives to be delivered; 

• a statement about how the department will 
increase the productivity of its operations. 

By this approach, the government is required to measure 
and improve the public sector's efficiency and thus raise 
its productivity, the quality and quantity of output 
delivered for the financial investment made. Each PSA will 
be monitored by the Government to meet every target. Should 
a target not be met there is no question of money being 
deducted from the budget for that department? Nor will 
additional funding over and above that already allocated be 
made available simply because a department is failing to 
meet its targets, but support and advice will be given by 

the committee .’^2 

To compliment the PSA, in 1998, the British Government 
also launched the Regulatory Impact Assessments (RIA) 
program to assess the likely outcomes of regulating in 
order to try to achieve the right balance between under¬ 
regulating, which may fail to protect the public, and over¬ 
regulating, which may create excessive bureaucracy. RIA 
identifies the costs and benefits of a policy proposal and 
the risks of not acting and is intended to inform the 
policy decision making process and communicate clearly the 
objectives, options, costs, benefits and risks of proposals 
to the public to increase the transparency of the process. 


'1 "Public Services for the Future: Modernisation, Reform, 
Accountability, Comprehensive Spending Review: Public Service 
Agreements 1999-2002," in The Stationery Office [database online] . 
Available from http://www.archive.official- 
documents.co.uk/document/cm41/4181/psa-4181.htm. 

12 ibid. 
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There are three stages on RIA. First, initial RIA 

carries out at the early policy development stage and helps 

identify areas where departments need more information. 

Second, partial RIA which builds on the Initial RIA should 

accompany the formal public consultation for the proposal. 

Finally, final RIA, which also on the analysis in the 

Partial RIA, updates it in the light of consultation and 

further analysis and information. The Final RIA should 

accompany legislation when presented to Parliament.’^3 

From the three stages above, it can be noted that the 

process of RIA, not only provides an evaluation on 

government policies but it is also accompanied by public 

involvement to obtain a soundness of analysis to the public 

interest. RIA has helped contribute to better policy making 

and reducing the costs to business and policy makers could 

pay more attention to some aspects of preparing RIAs which 

includes evaluating a range of options (including not 

regulating) and encouraging self-regulation where feasible. 

From an examination of RIAs in 13 departments and agencies, 

RIAs have been used in accordance with good practice and 

departments and agencies have applied the RIA process to 

good effect so that they achieve the five principles of 

good regulation: transparency, proportionality, targeting, 

consistency and accountability. For example: 

...consultation on the basis of the RIA on the 
proposed regime for the National Minimum Wage 
resulted in the adoption of a different method of 
implementation that avoided £150 million in 
unnecessary administrative costs to employers. And 
a RIA on new pesticides regulations showed that 


'3 "Good Practice: Regulatory Impact Assessments - Introduction 
to RIAs" [cited 2005]. Available from 
http://WWW.nao.org.uk/ria/ria_introduction.htm 
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employers would incur disproportionate costs from 
a new mandatory training requirement which were 
not justified by the benefits. So a non-regulatory 
option was adopted instead.’^4 

2. Canada 

Requirement to measure and report on performance in 
Canada has been started since the early 1970s. The effort 
has been strengthened by launching the Increased 
Ministerial Authority and Accountability (IMAA) in 1986, 
which demands increasing accountability for performance and 
allows departments to expand authority to reallocate 
resources within approved funding levels as well as to 
carry a portion of their capital funds into the next fiscal 
year budget. The MOU, which was being individually 
prepared and negotiated in each department, covers a three 
year period and consists of targets and an assessment of 
performance geared to the targets. 

After a public sector reform to improve the 
effectiveness and cost efficiency of all government 
programs in 1994, in 1996 the "Getting Government Right" 
initiative was introduced to modernize federal programs and 
has been confirmed as a significant change to a performance 
based management theory. Department and agencies were 
encouraged to identify result commitments and to measure 
and use performance information .^5 in April 1999, the 
Performance Management Programme was introduced to provide 
a framework for equitable performance management for 


'4 "National Audit Office Press Notice: Better Regulation - Making 
Good Use of Regulatory Impact Assessments," in NAO [database online]. 
[cited 2005], Available from http://www.nao.org.uk/pn/01- 

02/0102329.htm 
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federal executives and to reward the achievement of 
results. To assure the quality information, government 
departments adopt quality management as part of overall 
strategy and conduct internal audits and evaluations of key 
policies and programs. ^6 

Related to the high level objectives and type of 
measurement the measures are fairly comprehensive both in 
terms of inputs, outputs, outcomes, etc. and in coverage of 
the public service. However, concerns continue to be raised 
about the integration of results with management values; 
more comprehensive and accessible reporting; and 
development of targets etc. linking federal, provincial and 

local agencies. 77 

The agency which is responsible for managing the 
overall performance management in Canada is the Treasury 
Board Secretariat (TBS). Its representatives are placed in 
each department or agency. For reporting to Parliament, TBS 
shares its authority with federal bodies, within which 
Federal bodies provide Performance and Annual Reports to 
Parliament and TBS collates and reports specifically 
through the Annual Report. 


3. Australia 

Although until the mid-1980s, financial 
highly centralized, and spending departments 
opportunity or incentive to manage their 
interest in performance management dates back 


control was 
had little 
finances, 78 
to a report 


76 The Comptroller and Auditor General, Measuring the Performance of 
Government Departments,Anonymous , 2001. 

77 ibid. 

78 Schick, Budgeting for Results: Recent Developments in Five 
Industrialized Countries, 26-34. 
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on Australian Government administration in 1976 (this was 
called the Coombs' Report).^9 The development of performance 
measurement system in Australia was initiated in 1980's 
when the government launched two major reforms: the 
Financial Management Improvement Program (FMIP) and Program 
Management and Budgeting (PMB), which seeks to shift focus 
of budgeting from inputs used to the results achieved. A 
1983 government study found that 94 percent of its senior 
managers perceived financial management as merely spending 
the total allocated or controlling expenditures against 
appropriations . 80 Although FMIP was actively being promoted, 
it failed to balance the need for central direction with 
FMIP commitment to allow each department to set up its own 
management system. 

Program Management and Budgeting (PMB) is the key for 
identifying each organization's objectives and placing them 
into a framework within which targets can be set for 
managers at all levels.8i Schick continues by observing that 
the program structure arrays objectives in hierarchical 
order, with those at the lower levels more subject to being 
expressed in measurable terms than those at the higher 
levels. The development of Australia's PMB had become more 
complicated since many difficulties in defining aims and 
using performance indicators to measure results. 

79 Beryl A. Radin, "A comparative approach to performance management: 
Contrasting the experience of Australia, New Zealand, and the United 
States," International Journal of Public Administration 26, no. 12 
(2003): 1355. 

80 Australian Public Service Board and Department of Finance, Financial 

Management Improvement Program: Diagnostic study, February 1984; in 

Schick, Budgeting for Results: Recent Developments in Five 

Industrialized Countries, 26-34. 

81 Australian Public Service Board and Department of Finance, Financial 

Management Improvement Program: Diagnostic study, February 1984; in 

ibid. 
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In 1988, the federal government initiated a program 
for evaluative reviews of all policy areas and ensured that 
the results were considered in budgetary decision 
processes. In February 1993, The Management Advisory Board 
and Management Improvement Advisory Committee issued a 
paper called "Performance Information and the Management 
Cycle. It asserted that "performance information is being 
used in both program management and central decision¬ 
making, including in the budget and context; and the 
quality of performance reporting is improving."82 

To coordinate the delivery of public services, the 
federal government created 'Centrelink' in 1997, an agency 
functioned as the agency providing 'one-stop-shopping 
access to a range of government services. More recently the 
introduction of accrual based accounting and budgeting, so 
called Output Based Budgeting, has been aimed at providing 
high level performance information about outputs and 
outcomes, linked to more accurate resource accounting, so 
that broad 'cost benefit' judgments could be made.83 

Within the new approach, every agency is responsible 
for performance management. Changes to outputs and 
performance information are agreed with the relevant 
Minister and do not require clearance from the Department 
of Finance and Administration. Each department or agency 
issues information on resources consumed; outputs (price, 
quantity, quality) to be delivered and the outcomes to be 
contributed; actual outcomes and the contribution outputs 

82 Management Advisory Board Australia, 1993, Performance Information 
and the Management Cycle, AGPS, Canberra. In Radin, A comparative 
approach to performance management: Contrasting the experience of 
Australia, New Zealand, and the United States, 1355. 

83 The Comptroller and Auditor General, Measuring the Performance of 
Government Departments, 
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made to them; and finally the distinction between outcomes 
produced by agencies controlled by, and those administered 

on behalf of government.84 

Performance is reported publicly in individual agency 
Annual Reports to Parliament. Department of Finance and 
Administration has role in advising the development of 
performance measures and indicators. For the quality 
assurance of the information, agency audit committees are 
responsible for internal audit of performance information 
and no general requirement for audit and validation. 
However, the Australian National Audit Office (ANAO) 
performs value for money audits including resource use, 
delivery of outputs and outcomes, monitoring systems and 
compliance. Some systemic barriers to performance 
improvement in Australia are: 

a. No real link between appropriations and 
objectives. The federal government had a system 
of so called program budgeting but in reality 
had program reporting only. The budget process 
and the legislative appropriations process was 
largely determined by history. 

b. No clear relationship between the budgeted 
performance promises and the actual performance 
report in annual reports. 

c. Responsibility for program definition and 
reporting was devolved almost entirely to 
departments and agencies, with little 
legislative or executive oversight. This was 
reinforced by a cabinet system in which the 
specific cabinet minister charged with 
responsibility for a department was not likely 
to take "orders" from a peer.85 


84 The Comptroller and Auditor General, Measuring the Performance of 
Government Departments, 

85 Speech by Stephen Bartos, Australian Department of Finance, April 

26, 2000 In Radin, A comparative approach to performance management: 

Contrasting the experience of Australia, New Zealand, and the United 
States, 1355. 
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I. CHAPTER SUMMARY 

The GPRA has been a prominence best practice in 
performance based budgeting implementation. It is caused by 
a series of effort of performance oriented budgeting 
experiences from the first Hoover Commission, PPBS MBO, 
ZBB, and finally the GPRA. Albeit a lot of criticisms in 
the earlier of its implementation, GPRA has proved itself 
to be sound government performance evaluation tool facing 
enormous complexities in the public sectors. The important 
aspects that make GPRA successful are, first, it is enacted 
by the law so that it has supported by both President and 
Congress; second, it is included in the President 
Management Agenda in which President George W. Bush put 
this effort as one of the important program during his 
presidency; finally, 0MB has opened the involvement of the 
stakeholders such as GAO, Congress, agencies, academics, to 
provide any feedback towards its implementation and to 
enrich the development of the elements of GPRA. 

In the DOD, facing the complexities such as defense is 
particularly difficult area for which to develop measures 
of ultimate performance and it involves over $1 trillion in 
assets, budget authority of about $310 billion annually, 
and about 3 million military and civilian employees and 
these operations represents one of the largest management 
challenges within the federal government, the performance 
evaluation tools generated from the PPBES system has also 
provided a decision-making framework of budget formulation 
process and a basis of developing performance measurement 
system. Several pilot projects such as Defense Logistics 
Agency and the Defense Finance and Accounting Service can 
be set as best practice in defense business. 
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Flexibility is one of the key implementation in DOD, 
in which after having the framework consists of four 
dimensions of risk: force management, operational, future 
challenges, and institutional, DOD gave the defense 
agencies the option to use a balanced scorecard. Quick feed 
back has also been provided in implementing the GPRA. For 
example, facing some weaknesses in DOD strategies for 
achieving unmet performance goals in the future, DOD 
quickly replied by issuing a guidance to refine its 
approach to performance management and renamed the 
contracts as performance plans that intended to include a 
more comprehensive set of performance measures that align 
with agency strategic plans and department-wide goals set 
forth in DOD's risk management framework. 

The results of Outcome-Based Scrutiny tool from the 
Mercatus Center, George Mason University probably can be 
used as the third party judgment towards GPRA 
implementation, because, academically, it inform the public 
about the accountability perspective, transparency, public 
benefits, and leadership, of the GPRA implementation 
results produced by the agencies so that policymakers and 
citizens may make informed judgments about the agencies' 
results. This tool provides more sound on how government 
manages every Federal tax dollar and what level and quality 
of government service are returned to the taxpayer. 
Specifically, DOD still ranks as one of the lowest agencies 
which inform the performance to the public. 
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Ill. PERFORMANCE BASED BUDGETING 


A. INTRODUCTION 

This chapter lays out the literature review of 
Performance Based Budgeting (PBB), a prominent reform 
around the world. Section B reviews overview of PBB about 
its definitions and principle. Section C, D, and E provides 
the literature review of Performance Based Budgeting 
including strategic planning, performance measures and 
linking performance to budget. A summary is provided at the 
end of the chapter in Section F 

B. OVERVIEW 

Budgeting is essentially the allocation of resources; 
it inherently involves setting priorities. In its broadest 
sense, the budget debate is the place where competing 
claims and claimants come together to decide how much of 
the government's scarce resources will be allocated across 
many compelling national purposes. 86 

In traditional line item budgeting system, government 
expenditure is set-up based on line items or major object 
expenditure. In this system, budget control is done by 
stating the limits of spending on each item in the budget 
allocation. For the addition or deletion, incrementalism 
theory has played role. "Incrementalism" is widely held 
budgetary concept, which attributed to the works of Aaron 
Wildavsky and further developed by Richard Fenno. It was a 
bottom-up method of developing the budget through a series 


United States. General Accounting Office, Performance Budgeting: 
Current Development and Future Prospects, (Washington D.C.: U.S. 
General Accounting Office, 2003), 24. 

61 



of political adjustments that tended to increase the budget 
base in small controlled increments87 

In practice, line item budgeting with its 
incrementalism concept does not provide information about 
the reasons why government made the expenditures or even 
does not provide the aspects of efficiency and 
effectiveness of the programs. Decisions about future 
spending are taken without sufficiently examining the 
outputs or outcomes of the government programs. It can be 
said that without having an adequate examination of 
efficiency and effectiveness aspects, government 
accountability is still arguable. Here, performance based 
budgeting or performance budgeting can be used as a tandem 
to complement the line item budget system. 

However, it remains questionable whether after 
performance budgeting is implemented, the other budgeting 
process will not be used or whether the performance 
information is the only consideration in formulating the 
budget and allocating the resources. While GAO states that 
performance budgeting cannot replace the budget process as 
it currently exists, but it can help shift the focus of 
budgetary process as oversight activities by changing the 
agenda of questions asked in these processes ;88 Phillip G. 
Joyce, 89 on the other hand, states that agencies may replace 
traditional line-item budgets with a system that grants 
program managers greater flexibility in managing their 
resources but holds them accountable for achieving program 


O' Pitsvada and LoStracco, Performance budgeting—the next budgetary 
answer. But what is the question?, 53. 

88 United States. General Accounting Office, Performance Budgeting: 
Current Development and Future Prospects, 24. 

89 Joyce, Performance Based Budgeting, 597-619. 
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results. He continues by observing that even if government¬ 
wide decisions on resource allocation are not affected, 
agencies may find measures that are valuable for improving 
their management of a given level of resources, regardless 
of whether the use of those measures results in a 
significant shift of resources from on program or agency to 
another. Also, the government or particular agencies may 
use performance measures as a part of reports on their 
accomplishments. 

In general. Performance Based Budgeting is a set of 
process of translating strategic planning framework into 
performance measures in terms of input, output, and outcome 
to be tied into resource allocation in budgeting process. 
The concept of performance budgeting assumes that a 
systematic presentation of performance information 
alongside budget amounts will improve budget decision¬ 
making by focusing funding choices on program results. 
Melkers and Willoughby describes performance based 
budgeting as a more diverse set of requirements, blending 
various aspects of current public management trends, 
including outcome measurement, performance measurement 
systems, strategic planning and benchmarking.From 
definitions above, at least, three principles that must be 
established: strategic planning, performance measurement 

and linking performance to budget. 

90 United States. General Accounting Office. Accounting and 
Information Management Division, Performance budgeting [microform] 
fiscal year 2000 progress in linking plans with budgets,Anonymous 
(Washington, D.C. (P.O. Box 37050, Washington, D.C. 20013) : The Office, 
1999), 11, 2. 

91 J. E. Melkers and K. G. Willoughby, "Budgeters Views of State 

Performance-Budgeting Systems: Distinctions across Branches," Public 
administration review 61, no. 1 (2001) : 54-64. 
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c. 


STRATEGIC PLANNING 


Strategic planning is the process of deciding on 


objectives 

of 

the organization. 

on 

changes 

in 

these 

objectives, 

on 

the resources 

used 

to 

attain 

these 

objectives, 

and 

on the policies 

that 

are 

to 

govern the 


acquisition, use, and the disposition of these resources.92 
It blends futuristic thinking, objective analysis, and 
subjective evaluation of goals and priorities to chart 
future courses of action that will ensure the long-run 
vitality and effectiveness of the organization. 93 

Strategic planning is maybe such a new concept in 
public sector management which has been more familiar with 
the long term planning concept. The successful of this idea 
in private sector has brought public manager attention on 
how to apply it in the complex world of public sector. 
According to Robert B. Denhart, 94 the reasons for public 
organizations to undertake strategic planning efforts are 
many - to give clarity and direction to the organization, 
to choose from among competing goals and activities, to 
cope with expected shifts in the environment, and to bring 
together the thoughts and ideas of all participants in the 
work of the organization. 


92 Schick, Budgeting for Results: Recent Developments in Five 
Industrialized Countries, 26-34. 

93 Poister, Theodore H., and Gregory D. Streib, "Strategic Management 
in the Public Sector," in Performance Based Budgeting, ed. Miller, 
Gerald J., W. Bartley Hildreth, and Jack Rabin. Anonymous (Colorado: 
Westview Press, 2001), 283-305. 

94 Robert B. Denhardt, "Strategic Planning in State and Local 
Government," in Performance Based Budgeting, ed. Miller, Gerald J., W. 
Bartley Hildreth, Jack Rabin. Anonymous (Colorado: Westview Press, 
2001), 233-244. 


64 



A strategic planning is different with a long range 
planning. Robert B. Denhart^^ points three ways why 
strategic planning is different with a traditional long 
range planning. First, while traditional planning 
activities are concerned primarily with establishing goals 
or objectives over a period of time, but less concerned 
with specific steps that should be undertaken to achieve 
those goals, strategic planning consists of a series of 
action steps will develop as part of the planning process 
and activities guidance for organization in the immediate 
future in which it takes the future into account, but 
improves decisions in the present. 

Second is its attention to environmental complexity in 
which the organization is not assumed to exist in a vacuum, 
but rather both the organization's objectives and steps to 
achieve those objectives are seen in the context of the 
resources an constraints presented by the organization's 
environment.96 ype last distinction is that strategic 
planning, especially in the public sector, is a process 
that must involve many individuals at many levels of the 
organization. 

In designing the components of strategic planning, 
there are two approaches that can be used as benchmarking. 
First, GPRA has three components: mission statement, 
general goals and objectives, and approaches and strategies 


"5 Robert B. Denhardt, "Strategic Planning in State and Local 
Government," in Performance Based Budgeting, ed. Miller, Gerald J., W. 
Bartley Hildreth, Jack Rabin. Anonymous (Colorado: Westview Press, 
2001), 233-244. 

96 ibid. 
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to achieve the General Goals and Ob jectives. 9'^ Second, the 
Balanced Scorecard approach has seven components: mission, 
vision, core values, goals, strategy, strategic map and 
objectives. 98 

A mission statement is the core purpose of 
organization. It brings the agency into focus and explains 
why the agency exists and tells what id does.99A mission 
statement should be simple, clear, inspire and easy to 
understand. It clarifies the purpose of the organization 
and clearly articulates it to all stakeholders. In 

designing the mission statements, several key questions as 
following have to be answered: 

• Is the mission results oriented, and does it 
fulfill a public need? If not, how could the 
mission better focus on results? 

• Is the mission based on statute, and if so, 

does it cover all relevant statutes? 

• Are parts of the agency's functions or 

activities not covered in the mission 

statement? Why? 

• Are there developments (e.g. in technology or 
competition) that suggest the mission and 
corresponding legislation need to be revised or 
updated? 

• Is the agency's mission similar to those of 

other agencies and if so, has coordination 
occurred? Does unwarranted duplication of 

missions existed? 

• How is the agency's mission differentiated from 
those of other agencies with similar missions? 

Are there unique agency characteristics that 


91 United States. General Accounting Office, Agencies' Strategic Plans 
Under GPRA: Key Questions to Facilitate Congressional Review, 
(Washington D.C.: U.S. General Accounting Office, 1997), 37. 

98 Howard Rohm, Improve Public Sector Results with A Balanced 
Scorecard: Nine Steps to Success.Anonymous U.S. Foundation for 
Performance Measurement, 2002) 

99 United States. General Accounting Office, Agencies' Strategic Plans 
Under GPRA: Key Questions to Facilitate Congressional Review, 37. 
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give it an advantage in fulfilling its mission, 
such as location of field offices or staff 
expertise? loo 

Values are the time less principles that guide an 
organization and they represent the deeply held beliefs 
within the organization and are demonstrated through the 
day to day behaviors of all employees . loi Values must 
support the mission and help achieve organizational 
objectives. Patrick Lencionii02 suggests organizational 
values should be "aggressively authentic". Authentic in 
this context means developing values that are consistent 
with your organizational objectives, not writing something 
that would be well suited for the inside of a greeting 
card. Johnson and Johnson CEO Ralph Larsen stated that: 

".the core values embodied in our credo might be 

a competitive advantage, but that is not why 
would have them. We have them because they define 
for us what we stand for, and we would hold them 
even if they became a competitive disadvantage in 

certain situations". 

A vision statement provides a word picture of what the 
organization intends ultimately to become in the future and 
while mission statements are often abstract, the vision 
should contain as concrete of the desired state as possible 
ad provide the basis for formulating strategies and 
objectives. Peter Senge has observed: "Vision translates 


100 United States. General Accounting Office, Agencies' Strategic Plans 
Under GPRA: Key Questions to Facilitate Congressional Review, 37. 

101 Paul R. Niven, Balanced Scorecard Step-by-Step For Government and 
Nonprofit Agencies, (New Jersey: John Wiley and SOns, Inc, 2003), 3005. 

102 Patrick M. Lencioni, "Make Your Values Mean Something," Harvard 
Business Review, July 2002, pp.113-117. In ibid. 

103 ibid. 

104 ibid. 
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mission into truly meaningful intended results-and guides 
the allocation of time, energy, and resources. In my 
experience, it is only through a compelling vision that a 
deep sense of purpose comes alive". ^05 

According to Niven, to be effective, vision statements 
should be concise, balanced external and internal elements, 
appealing to all stakeholders, consistent with mission and 
values, verifiable, feasible and inspirational. He 
continues by providing ten questions to develop vision 
statement: 

• How would the world be improved or changed if 
we were successful in achieving our purpose? 

• What are the most important services that we 
should continue to provide, change, or begin to 
offer in the next three years? 

• What staffing and benefits changes do we need 
to implement to better achieve our purpose? 

• How will our elected officials or board of 
directors assist us in achieving our purpose? 

• What resource development (funding) changes to 
influence to better achieve our purpose? 

• What facilities and technology changes to 
implement to better achieve our purpose? 

• What infrastructure, systems, or communication 
changes do we need to implement to better 
achieve our purpose? 

• How could we more effectively or efficiently 
provide our service? If you could only make 
three changes that would significantly impact 
our ability to provide quality services to our 
clients/customers, what should these be? 

• What makes us unique? 

• What do our clients/customers consider most 
important in our provision of services? What do 
our clients/customers need form us? 

105 Patrick M. Lencioni, "Make Your Values Mean Something," Harvard 
Business Review, July 2002, pp.113-117. In Paul R. Niven, Balanced 
Scorecard Step-by-Step For Government and Nonprofit Agencies, (New 
Jersey: John Wiley and SOns, Inc, 2003), 3005. 
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General goals and objectives or strategic goals 
explain what results are expected form the agency's major 
functions and when to expect those results. Goals and 
Objectives should be concise, realistic, outcome-oriented 
and must be stated clearly in communicating what is be 
achieved and measured or assessed. In public sector, the 
goals and objectives should be linked to government policy 
decisions to create the basis for performance reporting. 
Several key questions that should be addressed in designing 
goals and objectives are: 

• Do the goals cover the major functions and 
operations of the agency? If not, what 
functions and operations are missing? Are the 
goals logically related to the mission? 

• Are the goals results oriented, such as to 

reduce crime or have fewer workplace accidents? 

Or, are they focused more on outputs, such as 
inspecting more workplaces? If so, why? 

• If the goals are not expressed in a 

quantitative or measurable form, are they 
expressed in a manner allow the agency and 
congress assessing whether goals are achieved? 

• Are all the agency's goals and priorities 
consistent with congress goals and priorities? 

When difference exist. Why do they exist, and 
can they be resolved? 

• Do the agency's goals appear similar to the 
goals in plans of other agencies that 
performing related activities? If so, are these 
sets of goals complementary or duplicative? 

• Are the goals targeted at results over which 
the agency has a reasonable degree of influence 
(may not apply to all agencies)?i07 


106 United States. General Accounting Office, Agencies' Strategic 
Plans Under GPRA: Key Questions to Facilitate Congressional Review, 37. 
101 ibid. 
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Strategies describe the 


means 


to achieve the 


objectives. It represents the broad priorities adopted by 
an organization in recognition of its operating environment 
and in pursuit of its mission, los qaO states that strategies 
help aligning an agency's activities, core processes, and 
resources to support achievement of the agency's strategic 
goals; mission and strategies should also outline how the 
agency will communicate strategic goals throughout the 
organization and hold managers and staff accountable for 
achieving these goals.109 Some key questions to be answered 
in designing strategy are: 

• How are the goals to be achieved? Are the 
strategies logically linked to the goals and 
the day-to-day activities of the managers and 
staff? Are they consistent with historical 
resource trends? 

• What steps will the agency take to align 
activities, core process, workforce, and other 
resources to support mission related outcomes? 

• What are the required resources, such as human, 
capital, and information? Are new regulations, 
flexibilities, user fees, or legislation 
required? 

• What steps is the agency taking to ensure that 
managers have the authority they need to 
achieve results? Are these strategies to hold 
managers accountable for results? Are there any 
strategies that focus on providing incentives 
for managers and other staff to achieve the 
goals ? 

• Do managers have the knowledge, skills and 
abilities to implement PBB? If not, what 
strategies are needed to develop the necessary 
capacity? 


108 Niven, Balanced Scorecard Step-by-Step For Government and Nonprofit 
Agencies, 3005. 

109 United States. General Accounting Office, Agencies' Strategic Plans 
Under GPRA: Key Questions to Facilitate Congressional Review, 37. 
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• Are technological advances necessary to 
successfully execute the strategies? If so, how 
likely are those advances? 

• What, if any, alternative strategies were 
considered? 

• Are there programs or activities that need to 
be eliminated. Created, or restructured to 
achieve the goals?iio 

In performance budgeting framework, McGill states that 
performance budgeting includes both a strategic framework 
and the mechanics of resource allocation in relation to 
performance. Ill The framework of strategic planning will 
direct agency in creating specific targets, performance 
measures, and mechanism of resource allocation in 
quantifiable data to provide meaningful performance 
information about program outcomes. 

These strategic contexts will help decision makers in 
budgeting process to rely not only based on qualitative 
performance information but also based on quantitative 
performance information. Here, it can be noted that 
strategic planning management plays an important role in 
PBB. A performance budget is one which presents the 
purposes and objectives for which funds are required, the 
cost of the programs proposed for achieving those 
objectives, and quantitative data measuring the 
accomplishment and work performed under each program. 112 


110 United States. General Accounting Office, Agencies' Strategic 
Plans Under GPRA: Key Questions to Facilitate Congressional Review, 37. 

111 Ronald McGill, "Performance budgeting," The International Journal 
of Public Sector Management 14, no. 4/5 (2001) : 376. 

112 Burkhead, Jess, 1956. Government Budgeting. New York: John Wiley. 
In Juul, DOD Implementation of the Government Performance and Results 
Act of 1993, 
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The important role of strategic management also 
provides an idea that PBB will only fit in the organization 
in long term organization goals. In practice, it happens 
because changes to the budget process are interconnected 
and include a shift from the traditional annual budget 
cycle to multi year frameworks as the basis for the annual 
budget process. Performance based budgeting can function 
somewhat like strategic budgeting since it will match 
within organization, the mission, goals and objectives, and 
strategies. Appropriate performance measures that 
accurately reflect agency efforts an accomplishment are 
essential for a successful planning and budgeting 
process . ii3 

In the short term, performance based budgeting is not 
likely to have any short-run effect on the way resources 
are allocated. The reason for that is because to the 
success of government policy can not be judged in short 
run. There might be delay between outputs and outcomes, or 
it may not be clear how much of a change in an outcome can 
be attributed to a program or policy. The design of 
performance indicators and establishing policy objectives 
requires a considerable amount of lengthy consensus 
building and performance indicators must remain stable over 
time to permit internal assessment of trends and to make 
the comparative judgments needed to assess progress, 

However, in practice, the underlying assumptions that 
PBB will fit in long term is hard to be accomplished 

113 ARMY WAR COLL CARLISLE BARRACKS PA and Ronald L. Harrell, 

Financial Management Transformation (*Performance* *Based* 

*Budgeting*), Anonymous , 2002), 35. 

114 Frank A. Schmidtlein, "Assumptions Underlying Performance-Based 

Budgeting," Tertiary Education and Management 5, no. 2 (1999) : 157. 
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because changes in policies and administration. Schmidtlein 
observes that: 


.these conditions commonly do not exist because 

government policy objectives typically are 
ambiguous and subject to frequent shifts. Another 
issues related to this assumptions is that as key 
officials come and go and political parties with 
differing values come to power, budgets are not 
followed exactly as prescribed when they were 
enacted because new political deal, 
circumstances, understanding arise for tactical 
and partisan advantage as well as for substantive 
reasons . ns 


D. PERFORMANCE MEASURES 

Performance budgeting assumes that future budget 
allocations will be influenced by performance-informed 
decisions (which presupposes a targeted strategic framework 
from which to make such decisions) it involves 
developing indicators of institutional performance and 
estimating the resources required to maintain or achieve 
selected levels of performance. Performance indicators 
has to be clear enunciated so that performance information 
can help managers to understand the performance of 
organization, and to set clear targets so that people 
understand the expected level of performance. 

According to Kouzmin, there has been evolution of 
determining the performance measurement. Whereas in the 
1980s the focus was on the "three Es", economy, efficiency, 
and effectiveness, in the 1990s attention has shifted to 


115 Frank A. Schmidtlein, "Assumptions Underlying Performance-Based 

Budgeting," Tertiary Education and Management 5, no. 2 (1999): 157. 

116 McGill, Performance budgeting, 376. 
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quality and consumer satisfaction. ns xhe quality and 
consumer satisfaction, which emphasis the efficiency and 
effectiveness of allocating resources, has left out the 
economy perspective. Especially in developed countries, 
economy perspective is assumed as anti-thesis for having a 
good quality because lowest prices are assumed as bad 
quality. 

However, in accordance with the evolution of 
performance measurement as mentioned above which according 
to government worldwide is intended to enhance the quality 
of delivery of services, people are still questioning the 
adequacy of government service quality. Although the cost 
of services is rising due to lack of economy perspective, 
the quality is inappropriate. Government accountability is 
still big question that can be answered by the government. 
Therefore, excluding the economy perspective in determining 
the cost of services is not the answer. 

Performance measures will provide information on what 
traditional "incremental" budgets or financial statements 
concerned with the results of the services delivered by the 
government. The task of measuring the performance in public 
sector is complex and difficult due to its intangible 
outputs, quality variations, and complex environment. The 
process of government to measure and judge its performance 
by determining how effectively and efficiently taxpayer 
resources are being used for the delivery of services and 
the administration of programs is critical to enhance 
operational accountability of governments. 

118 Alexander Kouzmin et al., "Benchmarking and performance measurement 
in public sectors Towards learning for agency effectiveness," The 
International Journal of Public Sector Management 12, no. 2 (1999): 
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Performance measures can be classified into five 
categories outputs, outcomes, economy, efficiency and 
effectiveness. Outputs measure gauge the amount of services 
delivered to citizens and allow one to examine what 
services and the quantities of services an entity is 
providing.119 It refers to the internal activities of a 
program (i.e., the products and services delivered) and 
describes the level of activity that will be provided over 
a period of time, including a description of the 
characteristics (e.g., timeliness) established as standards 
for the activity. 120 

Outcomes are what an organization is trying to 
achieve. It is the impact sought or expected by government 
in a given policy arena, which has focus on change and 
consequences: what effect government can have on the 
community, economy and/or national interest. 121 Outcomes 
are important to the effectiveness of programs/activities 
but may be more difficult to measure and assess than the 
inputs and outputs and it will often be influenced by 
external factors and may require long-term rather than 
short-term assessment .122 Outcome measures are the ultimate 
results that benefit the public. One of the biggest 
problems in designing performance measures is how to define 


119 "Performance Measurement: Connecting Strategy, Operations and 
Actions," [cited 2005] . Available from 

http://64.233.179.104/search?q=cache:awZcgotnKlUJ:www.idii.com/wp/donov 
an_perform.pdf+%22Performance+measurement:+connecting+strategy,+operati 
ons+and+actions%22&hl=en&client=firefox-a. 
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122 Auditing Standards Committee, International Organization of 
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the outputs and outcomes in an organization. Sometimes it 
is hard to differ between the two for public managers due 
to its intangible characteristics. An example of output and 
outcomes measures is as shown here: 123 


Table 6: Example of Out 

puts and Outcomes measures 

Outputs 

Outcomes 

Number of housing units 
rehabilitated. 

Increases in equity (property 
value) of rehabilitated houses 
for low-income families as a 
result of targeted assistance. 

Number of businesses 
assisted through loans and 
training. 

Percent of businesses that 
remain viable 3 years after 
assistance. 

Number of people served by 
water/sewer projects. 

Increased percent of people 
with access to clean drinking 
water. 

Number of acres of 
agricultural lands with 
conservation plans. 

Percent improvement in soil 
quality; dollars saved in 
flood mitigation. 


In establishing the outcome measures, in some cases, 
it is hard for agencies and departments to design the 
outcome measures. In this matter, agencies can address this 
problem by developing intermediate outcomes, that is, 
partial outcomes which are achieved within a shorter time 
frame and this would enable the better identification of 
target groups and provide a more appropriate basis for the 
development output.124 Following example can be use as a 
drilling down to get to the right outcome measure for a job 

training program: 125 


123 Office of Management and Budget, 0MB's Program Assessment Rating 
Tool: FY 2006 PART Instructions, 
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Table 7: Performance Measures on Job Training Program 


o Dollars appropriated to the program 
o Number and size of grants 

Why do these matters? What do they buy? 

Inputs: 

Funding (Federal and 
perhaps State and 
local) 

o Number of classes attended by program 

Outputs: 

participants 

Products (e.g., 

o Number of people trained 

classes taught. 

Why do these matters? What result do they 

service delivered. 

produce? 

participants serviced) 

o Number of people with useful skills 

Intermediate outcomes: 

o Number of people who get a job after 

(e.g., new knowledge. 

leaving the program 

increased skills. 

Why do these matters? Is this the result the 
public is seeking? 

changed behavior) 

o Number of program participants who remain 
employed for a specified time and increase 
their earnings 

o Number of people who are self-sufficient 

Program outcome 

Societal outcome 


An economy measure looks at the costs of acquiring the 
inputs which compare the resources available to be 
allocated and those used as input for performing an 
activity. Economy, which occurs where equal-quality 
resources are acquired at lower prices, is minimizing the 
cost of resources used for an activity, having regard to 
the appropriate quality and focus on the cost of the inputs 
and processes . 126 Economy not only means to have the lowest 
price but also to have cost reduction in allocating the 
resources to save the taxpayer money. 

An efficiency measure looks at whether the maximum 
outputs for the inputs that go into the process have been 
achieved. Efficiency is 'the relationship between the 
output, in terms of goods, services or other results, and 
the resources used to produce them and it exists where the 

126 Auditing Standards Committee, International Organization of 
Supreme Audit Institutions, Auditing Standards, 
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use of financial, human, physical and information resources 
is such that output is maximized for any given set of 
resource inputs, or input is minimized for any given 
quantity and quality of output."127 Sound efficiency 
measures capture skillfulness in executing programs, 
implementing activities, and achieving results, while 
avoiding wasted resources, effort, time, and/or money.128 

According to 0MB, the best efficiency measures capture 
improvements in program outcomes for a given level of 
resource used, for example, a program that has an outcome 
goal of "reduced energy consumption" may have an efficiency 
measure that shows the value of energy saved in relation to 
program costs. However, it may be difficult to express 
efficiency measures in terms of outcomes, because in some 
cases, acceptable efficiency measures could focus on how to 
administer the program better.129 OMB continues by stating 
that meaningful efficiency measures consider the benefit to 
the customer and serve as indicators of how well the 
program performs, for example, reducing processing time 
means little if error rates increase. A balanced approach 
is required to enhance the performance of both variables in 
pursuit of excellence to customers. In these instances, one 
measure (e.g., increase in customer satisfaction) may be 
used in conjunction with another complementary measure 
(e.g., reduction in processing time). 130 


127 Auditing Standards Committee, International Organization of 
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Effectiveness measures gauge the quality and 
productivity in the delivery of a service to citizens and 
it has characteristics including citizen's evaluations, 
timeliness, readiness, conditions and accuracy. i3i it looks 
at whether the outputs of the program lead to the desired 
outcomes. According to INTOSAI, 132 effectiveness is the 
extent to which objectives are achieved and the 
relationship between the intended impact and the actual 
impact of an activity and it addresses the issue of whether 
the program/activity has achieved its objectives. INTOSAI 
also states that when focusing on effectiveness, it is 
important to distinguish between the immediate outputs (or 
products) and the ultimate impacts (or outcomes). 

In designing performance measures in accordance with 
agencies goals and objectives as well as their outcomes, it 
is possible that among agencies have the same goals, 
objectives and outcomes. Crosscutting is a performance 
measurement method to evaluate interdependent government 
programs which have similar purposes, goals, and outcomes. 
This could be an internal crosscut, programs within the 
same agency, or external crosscut, programs across multiple 
agencies. Crosscutting would identify exemplary goals and 
practices, common measures of performance, possible 
tradeoffs in management and budget decisions, and 
opportunities for better coordination among programs. 133 


131 Donovan, Performance Measurement: Connecting Strategy, Operations 
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Crosscutting can be designed by having a joint working 
among departments and agencies which have similar outcome. 
In choosing the most appropriate measurement method for 
encouraging joint working, factors to be considered 
include: (1) the number of departments and other 
stakeholders involved; (2) the degree of stakeholder 
interpretation of national objectives needed for cost 
effective pursuit of objectives; (3) the priority accorded 
the objectives; and (4) the costs and burdens of given 
approaches to setting targets and monitoring progress.134 

In United States, crosscutting is designed in PART 
assessment. Unlike a combined PART, a crosscut would still 
examine programs in individual PARTS. By having the 
crosscut, individual assessment of programs in individual 
PART and the recognition of the distinctive features among 
programs still can be evaluated. However, according to 0MB, 
the following common themes could be used to compare the 
programs in a meaningful crosscut: 

• What is the target population for each program? 

• What products and services are provided (common 
output measures as appropriate)? 

• How well products and services are provided 
(common output efficiency measures as 
appropriate)? What impact have the programs 
achieved (common outcome measures and common 
outcome efficiency measures as appropriate) ?135 


134 xhe Comptroller and Auditor General, Measuring the Performance of 
Government Departments, 

135 Office of Management and Budget, 0MB's Program Assessment Rating 
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After the performance measures are gauged, it is 
important to compare the results against established 
standards. Standards can be factors such as the level and 
quality of client services and are set with the aim of 
defining the appropriate level of performance expected to 
be delivered. Standards or benchmarks could be set by 
professional organizations, the performance of other 
government entities, composite indices of select government 
entities, or pre-set targets or previous performance of 
entity. 136 

However, a comprehensive set of benchmarked results in 
the public sector have not been embraced by the government 
it self or professionals. An alternative to replace the 
benchmarks or standards is to use performance targets and 
baselines just like United States has in GPRA. Baselines 
are the starting point from which gains are measured and 
targets are set and the baseline year shows actual program 
performance or prior condition for the given measure in a 
specified prior year. 137 Targets refer to improved levels 
of performance needed to achieve the stated goals and they 
must be ambitious (i.e., set at a level that promotes 
continued improvement) and achievable given program 
characteristics; each target should be quantifiable and 
must have a timeframe (e.g., year(s) in which the target 
level is to be achieved) . In most instances, these targets 
should be quant if iable . i38 


136 Bourne, Mike and John Mills, "Designing, Implementation and 
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E. 


LINKING PERFORMANCE TO BUDGET 


Finally, the process that must be established is 

linking performance to budget. The performance measurement 
is tied directly to the budget process comprehensively and 
elaborates it to evaluate organization's performance in 

terms of outcome, output, and impact. It will give a 

meaningful indication of how the dollars are expected to 
turn into results and it can be distinguished from an 

Object Class budget in a fundamental way because while in 
the object class budget what each dollar will be spent 
would be showed, in the performance budgeting, inherently, 
what each dollar will accomplish also would be showed. 139 

In integrating performance into the budget, 
performance measures should capture the essence of the 
policy objective and the contribution of program activities 
to the outcome. Performance information could help 

policymakers address a number of questions, such as whether 
programs are contributing to their stated goals, well 
coordinated with related initiatives at the federal level 
or elsewhere, and targeted to those most in need of 
services or benefits. It can also provide information on 
what outcomes are being achieved, whether resource 
investments have benefits that exceed their costs, and 
whether program managers have the requisite capacities to 
achieve promised results, no 

In this stage, at least there three important point 
that has to be considered in how to link performance to 
budget resources. First is the budget alignment in which 

139 "Performance Based Budgeting: Overview," [cited 2004]. Available 
from http://www.john-mercer.com/pbb.htm 

140 United States. General Accounting Office, Performance Budgeting: 
Current Development and Future Prospects, 24. 
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structural and format changes to congressional budget 
justifications, and in some cases, appropriations accounts 
to better align resources with programs and performance. 
The need to change the account structure is important 
because some performance goals cut across multiple program 
activities and budget accounts or a single program can 
consist of multiple accounts. Budget restructuring efforts 
have sought to help reframe budget choices and to focus 
decisions more on the expected results of budget resources 
and less on inputs or line items.i4i 

The second aspect is costing method. There are two 
methods that can be used: full cost and activity based 
costing (ABC) . Full cost is the way in which changes to 
certain budget resources are distributed or measured to 
better reflect where and when resources are consumed. 0MB 
defined full cost as the sum of all budget resources used 
by an agency to achieve program outputs. 142 yhe budget 
resource can include not only traditional elements of 
costs, such as salaries and expenses, procurement of goods 
and services, grants, and transfers but also the cost of 
all support services and goods used and provided centrally 
which can be including accruing retiree pension and health 
benefits. Budgeting for the full cost of resources where 
they are used, making budget program and activity lines 
more parallel with outputs, and, where useful, improving 
alignment of budget process.i43 


141 United States. General Accounting Office, Performance Budgeting: 
Efforst to Restructure Budgets to Better Align Resources with 
Performance (Washington D.C.: U.S. General Accounting Office, 2005), 
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Full cost is generally viewed as including both direct 
costs, costs that can be specifically identified with a 
cost object, such as an output, materials, and supplies 
used in work, office space, and equipment and facilities 
that are used exclusively to produce the output, 144 and 
indirect cost, costs of resources that are jointly or 
commonly used to produce two or more types of outputs but 
are not specifically identifiable with any of the outputs 
such as general administrative services, general research 
and technical support, security, rent, and operations and 
maintenance costs for building, equipment and utilities, 

Activity Based Costing (ABC) is used to identify, 
describe, assign costs to and report on agency operations; 
it identifies opportunities to improve business process 
effectiveness and efficiency by determining the "true" cost 
of a product or service. 146 abC principles are used to focus 
management attention on the total cost to produce a product 
or service and as the basis for full cost recovery. The 
process ABC, started by determining the cost activity pools 
based on defined activities, assigning the costs the 
activity cost pools, calculating the activity rates and 
finally assign the costs to cost objects using the activity 
rates and activity measures, will provide users with better 
understanding of the cost information for strategic 
decisions. 


144 OMB, Circular No. A-11 (July 2002), p. 221-5 in ibid. 

145 OMB, Circular No. A-11 (July 2002), p. 221-5 in ibid. 

146 "Activity-Based Costing: Accounting for Operational Readiness," in 
The Learning Source OSD Comptroller iCenter [database online]. [cited 
2005]. Available from http://www.ditc.mil/comptroller. 
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ABC can become the basis for establishing performance 
measures; targeting cross-functional for improvement 
analysis; determining the gap in performance that may 
require reengineering; benchmarking process improvement to 
identify proven practices within organization and with 
external competitors and other organizations; and 
developing performance based budgeting plans.147 

The third important aspect is designing the assessment 
tool. The Performance Assessment Rating Tool (PART) in GPRA 
is maybe one of the best practices that can be used in this 
matter. PART is basically generated from the strategic 
goals and developed to be specific operational performance 
goals. Performance goals are expressed in a measurable and 
attainable objective to be matched with actual achievement 
in terms of outputs of outcomes. 

According to OMBUS^ the PART requires two types of 
performance goals. First, long-term performance goals 
address performance that is generally several years or more 
in the future which includes two basic types of long-term 
goals, an annual performance goal in the future (e.g., 
tornado warning times in 2008, or unit costs of an activity 
in 2010); and the cumulative effect of annual activities 
(e.g., development of an AIDS vaccine by 2010) . Second, 
Annual performance goals should be stated in yearly 
increments For example, for the weather program, an annual 
performance goal might include the same performance measure 
(advance warning time), but a less ambitious target (e.g.. 


147 Tracey G. Amos, Cynthia A. Paolillo, and Denise A. Joseph, 

"Enhancing CFO, GMRA & GPRA implementation with activity based 
management," The Government Accountants Journal 46, no. 1 (1997): 28. 
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15 minutes average warning time in 2005) due to less 
widespread use of advanced technologies. 

PART which contains 25 questions (See Appendix B for 
detailed PART Questions) is divided into four sections with 
specific weight for each section to create the numerical 
scores in the four management categories as following: 

Table 8: Overview of Sections of PART Questions 

Section Description Weight 


I. Program Purpose & 
Design 


To assess whether 


• The purpose is clear, and The 
program design makes sense 
To assess whether the agency 
sets valid programmatic 

II. Strategic Planning 

•Annual goals and Long term 
goals 

To rate agency management of the 
program including 

rogram Management 

•Financial oversight, and 

Program improvement efforts 
To rate program performance on 
ogram Results/ goals reviewed in 

ountability •The strategic planning section, 

and Through other evaluations 
Source: From GAO Report, GAO-04-174, p. 9.149 


Program Management 


IV. Program Results/ 
Accountability 


To aid standardization, the questions that comprise 
the PART are generally written in a Yes/No format. The 
part's instructions (see Appendix B for detailed 
Instructions of the Program Assessment Rating Tool) make a 
clear point that to be awarded a yes answer; the program 
should have a high standard of performance with a high 
level of confidence. However, according to GAQi^o^ the 


14y United States. General Accounting Office, Results-Oriented 
Government: GPRA Has Established a Solid Foundatuon for Achieving 
Greater Results, 270. 

150 United States. General Accounting Office, Performance budgeting 
[electronic resource] : observations on the use of 0MB's program 
assessment rating tool for the fiscal year 2004 budget, 
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format of yes and no response has resulted in 
oversimplified answers to some questions. Some parties felt 
that the format did not adequately distinguish between the 
performances of various programs. To solve the problem, in 
spring 2002, 0MB revised the PART to include four responses 
by adding "small extent" and "large extent" to the original 
two choices "yes" and "no") . It can be seen from standard 
of a yes in fiscal year 2006 PART instruction: 

The PART holds programs to high standards. Simple 
adequacy or compliance with the letter of the law 
is not enough. Rather, a program must show it is 
achieving its purpose and that it is well 
managed. The PART requires a high level of 
evidence to justify a Yes response. Sections I 
through III are scored in a Yes/No format. In 
Section IV, a four-level scale (Yes, Large 
Extent, Small Extent, and No) permits answers to 
reflect partial achievement of goals and evidence 
of results. Answers must be based on the most 
recent credible evidence.i5i 


To convert the PART scores into qualitative ratings, 
0MB uses the following scoring bands: effective (85-100), 
moderately effective (70-84), adequate (50-69), and 
ineffective (0-49). If the measures lack baselines and 
performance data and the program does not have performance 
measures that have been agreed-upon by 0MB, a rating of 
Results Not Demonstrated (RND) is given. Gilmour and Lewis 
state that the last category is not intended to reflect 
well or poorly on a program. Rather, the "the results not 
demonstrated" grade reflects OMB's judgment that the 
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existing measures for a program do not allow an assessment 
of the program's ef feet iveness . 152 

The outputs of PART then will be transferred into a 
scorecard. The 0MB's Executive Branch Management Scorecard 
contains six "Core Criteria". The highest rating of "green" 
a federal agency must satisfy all six criteria. A rating of 
"yellow" is awarded if the agency meets six similar 
criteria that establish a lower level of achievement. 
However, if any one of six defined "Conditions" is found to 
exist, the agency will automatically be given the lowest 
rating of "red." The detail of the scorecard core criteria 
can be seen in Appendix C. 

By having performance measures linked to budget, if 
PBB could provide what government programs have good 
results or poor results, how the government will be 
encouraged? Will government be rewarded or punished? Under 
performance budgeting, people should not expect that good 
results will always be rewarded through the budget process 
while poor results will always have negative funding 
implications .153 However, decisions on funding levels may be 
intended either to reward high achievement or to penalize 
inadequate achievement .154 gaO continues by observing that 
viewing performance budgeting as a mechanistic arrangement- 
a specific level of performance in exchange for a certain 

152 John Gilmour B. and David Lewis E., "Assessing Performance 
Assessment for Budgeting: The Influence of Politics, Performance, and 
Program Size in FY 2005," Journal of Public Administration Research and 
Theory (2004) 

153 United States. General Accounting Office, Performance Budgeting; 
Current Development and Future Prospects, 24. 

154 Schmidtlein, Assumptions Underlying Performance-Based Budgeting, 
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amount of funding-or in punitive terms-produce results or 
risk funding reductions-is not useful. Such mechanistic 
relationships cannot be sustained. Rather than increase 
accountability these approaches might instead devalue the 
process by favoring managers who meet expectations by 

aiming low. 155 

McNab and Melese observe that if the GPRA is to create 
an explicit linkage between budget appropriation (and 

obligations) and the outcomes generated by public 
expenditure, it must assist in the creation of an 

institutional environment that rewards efficiency, 
transparency, and the prompt, concise, and accurate 
reporting of costs, outputs, and outcomes.156 In the absence 
of such an environment, departments may respond to the 
current incentive structure by "gaming" their performance 
reports to present their activities in terms designed to 
maximize their budget. 157 

GAO claims that PBB can play an important role in 

creating incentives in a pay-for-performance system, for 
example, once a sufficient level of result is achieved, 
managerial bonuses might be linked to maintaining that 
level or service, while decreasing the program unit costs; 
this creates a particularly strong incentive fro managers 
not to spend every program dollar where it is not necessary 

155 United States. General Accounting Office, Performance Budgeting: 
Current Development and Future Prospects, 24. 

156 McNab and Melese, Implementing the GPRA: Examining the Prospects 
for Performance Budgeting in the Federal Government, 73-95. 
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for achieving the desired result. 158 This is also the reason 
why public managers shifting their attention from 
traditional "incremental" line item budgeting to 
performance based budgeting. 

F. CHAPTER SUMMARY 

Performance based budgeting can be used as a 
supporting tool for the line item budgeting since PBB 
provide the performance information that can not be 
provided by line item budgeting. Performance Based 
Budgeting basically is a set of processes translating the 
strategic planning framework into performance measures in 
terms of input, output, and outcome to be tied into 
resource allocation in budgeting process. Three principles 
that must be established in PBB are strategic planning, 
performance measurement and linking performance to budget. 

Albeit strategic planning is still new for the public 
sector, several reasons such as to give clarity and 
direction to the organization, to choose from among 

competing goals and activities, to cope with expected 
shifts in the environment, and to bring together the 

thoughts and ideas of all participants in the work of the 
organization has shifted the public sector manager to 
utilize it in their organization. In designing the 
components of strategic planning, there are two approaches 
that can be used as benchmarking. First, GPRA has three 
components: mission statement, general goals and 

158 "Congressional Testimony; Statement of John Mercer before the 
Subcommittee on Government Efficiency, Financial Management, and 

Intergovernmental Relations U.S. House Representative, June 19, 2001; 

The Results Act: Has it Met Congressional Expectations?" [cited 2004] . 
Available from http://www.john-mercer.com/library/jn_testimony.pdf. 
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objectives, and approaches and strategies to achieve the 
General Goals and Objectives. Second, the Balanced 
Scorecard approach has seven components: mission, vision, 
core values, goals, strategy, strategic map and objectives. 

In designing the performance measurement system, 
although there has been evolution of determining the 
performance measures in which the focus has been shifted 
from in the 1980s on the "three Es", economy, efficiency, 
and effectiveness, to in the 1990s on quality and consumer 
satisfaction, the factor of the wealthy of a country can be 
used in determining which measures that will be taken. 
While developed countries which have enormous resources to 
be allocated for government budget might use the quality 
and consumer satisfaction which emphasis the efficiency and 
effectiveness assessment, developing countries which have 
limited resources might not leave the economy perspective 
to obtain cost reduction in government budget. 

In linking performance to budget, the performance 
measures not only capture the essence of the policy 
objective and the contribution of program activities to the 
outcome but also provide information on what outcomes are 
being achieved, whether resource investments have benefits 
that exceed their costs, and whether program managers have 
the requisite capacities to achieve promised results. Three 
important aspects that have to be considered in how to link 
performance to budget resources are: budget account changes 
to better align resources with programs and performance; 
full cost or ABC to better reflect where and when resources 
are consumed; and the assessment tool in which. The 
Performance Assessment Rating Tool (PART) in GPRA is can be 
used as the best practice. 
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IV. A MODEL FOR INDONESIAN DOD 


A. INTRODUCTION 

This chapter provides a model of performance based 
budgeting for Indonesian DOD/TNI by incorporating the 
knowledge and information from the Chapter II and Chapter 
III. Section B presents a short overview of budgeting 
process in Indonesia. Section C discusses the progress of 
the implementation of Performance Based Budgeting in 
Indonesia as well as several comparisons between the 
current progress and United States Experiences in GPRA. 
Section D provides a cascade model of performance based 
budgeting process for Indonesian DOD/TNI. A summary is 
provided at the end of the chapter in Section E 

B. OVERVIEW 

The 1945 Constitution of the Republic of Indonesia 
recognizes six state institutions: (1) The People's 

Deliberative Assembly or Majelis Permusyawaratan Rakyat 
(MPR); (2) The President; (3) The Supreme Advisory Council 

or Dewan Pertimbangan Agung (DPA ); (4) The House of 

People's Representatives or Dewan Perwakilan Rakyat (DPR); 
(4) The Audit Board of the Republic of Indonesia or Badan 
Pemeriksa Keuangan (BPK); (5) The Supreme Court or Mahkamah 

Agung (MA). 

The hierarchical position of the six institutions is as 
following: 
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Figure 1: Hierarchical Position of Six Supreme Institutions 

in Indonesia 



Notes: 

1. MPR: The People's Deliberative Assembly or Majells 
Permusyawaratan Rakyat; 

2. DPA: The Supreme Advisory Council or Dewan 

Pertimbangan Agung; 

3. DPR: The House of People's Representatives or Dewan 
Perwakilan Rakyat; 

4. BPK: The Audit Board of the Republic of Indonesia or 
Badan Pemeriksa Keuangan; 

5. MA: The Supreme Court or Mahkamah Agung. 

From the above figure, it can be seen that the highest 
constitutional body is the MPR which is selected every five 
years. Half of the MPR members are assigned as member of DPR 
which is constitutionally has legislative authority. This 
condition also gives strong power to DPR. However, it can be 
dissolved by President. If the President exceeds his 
authority the DPR can convene the MPR to bring the President 
to account. The President as the principal executor of state 
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government is not responsible to DPR but to MPR except for 
the law enactment and state budget approval in which 
President needs DPR approval. BPK is responsible for 
auditing the government accountability in using the state 
funds and delivering the report to the DPR. Ministries and 
Heads of Agencies are appointed by the President and are 
responsible to the President. For government structure, the 
Indonesian Government has a three tier government 
structures; state, regional, and local. 

Indonesian budgeting system has evolved dramatically 
for the past two years, especially with enormous pressures 
from investors to better manage financial management and 
enhance government accountability. Lessons learned from the 
fragile system before the monetary and economic crisis in 
1998 have provoked the stakeholders to demand changes 
towards how the government runs its business. 

In the past, the financial management system was linked 
to the Indische Compatabiliteits Wet (ICW) inherited from 
Ducth Colonialism, and further amended from time to time and 
called the Indonesian Financial Administration Act. 
Realizing the inadequacy of the ICW, gradually the 
government has enacted several laws related to financial 
matters. The important law that has become the starting 
point of the willingness to change is the State Finance Law 
No. 17/2003. Before the enactment of this law, the budgeting 
system was the traditional line item budgeting system. The 
state budget was prepared on the basis of the Five Year 
Development Plans (REPELITA) in accordance with the General 
Guidelines of State Policy (GBHN) . GBHN was a long term 
national development plan for a twenty-five year period and 
was updated every five years. 
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The budget appropriation was prepared on an annual 
basis. The budget estimates were prepared annually and 
submitted to the DPR for approval. Any policy and provision 
changes on the approved budget need DPR approval. The budget 
bill, having approved by the DPR and signed by the 
President, comprised of state revenues and state 
expenditures. While state revenues consisted of two 
classifications. First was routine revenues, revenues 
internally generated from taxes revenues and non taxes 
revenue (PNBP); second was development revenues, revenues 
received in foreign loans. State expenditures also consisted 
of two classifications: routine expenditures and development 
expenditures. They were allocated into sectors and sub 
sectors. 

C. PERFORMANCE BASED BUDGETING IN INDONESIA 

Before the government reform which was initiated after 
the 1998 crisis, there was no performance evaluation tool 
provided by the government to assess the execution of its 
tasks and activities. Government had a strong power over 
all other supreme institutions in the country. The only 
performance evaluation was DPS, a performance evaluation 
for civil servant. However, this tool was still 
inappropriate and consisted of many unclear and biased 
measures. 

After the 1998 crisis, feeling pressure from investor 
countries, a series of action was taken. The first effort 
was by launching the President Instruction No. 7, 1999 that 
required departments, agencies and regional and local 
government to prepare the Accountability and Performance 
Reports, AKIP report (LAKIP-Laporan Akuntabilitas Instansi 
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Pemerintah) . However, 

it 

has 

a serious 

flaw in that 

the 

agency or department 

is 

free 

to assign 

a weight to 

each 

activity performed 

and 

the 

danger is 

that the agency 


attempt to engineer the weight in order to search the 
"best" result, regardless the improvement and furthermore, 
this has been perceived as a useless exercise since there 
is no follow-up or utilization of the LAKIP data.159 

The second effort is the involvement of investor 
countries through Non Governmental Organization and 
consultants is to assist local governments in preparing a 
new result oriented budget, performance budgeting. This 
effort has been taken in accordance with the 
decentralization policy which was initiated in 1999. It is 
clearly stated in the 8th paragraph of Government 
Regulation (PP) No. 105/2000 that local government should 
follow performance budgeting in preparing the annual 
budget. However, the biggest challenge to implement the 
performance budgeting in local government is lack of 
understanding of the concept of performance budgeting. 
Several other issues faced by local government are lack of 
general guidelines for local planning; delayed release of 
new guidelines for local budgeting and financial 
management; vague expenditure planning; and lack of 
qualified local planning and budgeting officials.160 

For central government, by the enactment of two laws, 
the Law No. 17/2003 and the National Development Planning 
System Law No 25/2003 gradually the government transforms 
how they run business. The National Development Planning 

159 Ahmadi H. Ringoringo, Performance Budgeting: The Impact of Pubic 
Expenditure Policy to Local Government, Anonymous , Jakartaed. Project 
497-0357/104-000, USAID-funded Partnership for Economic Growth, 2002) 

160 ibid. 
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System Law No. 25, 2003 was enacted to shift the national 
planning from GBHN system to the new planning system so 
called PROPENAS (National Planning Program). By this new 
law, the planning system will initiate strategic planning by 
incorporating a twenty-year period of long term planning, 
divided into four medium term planning (five years) and 
short term annual planning. The PROPENAS provides directions 
for development strategy at all levels of government, which 
determines sectoral allocations of development spending for 
central and regional levels. Strategic development plans of 
central line ministries and agencies (RENSTRA) are produced 
based on PROPENAS, accompanied by the annual national 
development plan (REPETA) defining priorities in the 
national development budget. 

The State Einance Law No. 23 2003 provides significant 
changes for state, regional and local budgeting systems. 
Ministry of Einance has cooperated with other related 
agencies and consultants to prepare the manual and 
additional regulation to cover the implementation of 


performance budgeting. 

It 

is expected that 

the 

full 

implementation will 

be 

started in EY 2006. There 

are 

three 

important changes 

for 

the 

budgeting process. 

The 

first 

important aspect in 

this law 

is the requirement 

for 

state 


and local governments to implement results oriented 
budgeting by integrating the performance accountability 
system into the budgeting system. The second important 
aspect is the requirement to unify the budget 
classification, called unified budget. This change will 
eliminate the previous budget classification of routine and 
development budgets which created redundancy, inefficiency 
and even fraud in departments and agencies. The third 
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important aspect is Medium Term Expenditure Framework (MTEF) 
to enhance the inter-correlation between the planning and 
budgeting process and to obtain more rational and strategic 
resources allocation. It is also mentioned that in future 
year government will provide Government Accounting Standard 
so that government financial statements can be more 
transparent and accountable. 

In the State Budget for FY2005, the government has made 
some progress and adjustment in accordance with the Law no. 
23/2004. The important move is that government has 
reclassified the budget accounts. The purposes of the budget 
reclassification are (1) to avoid budget duplication due to 
lack of policy to distinguish between operational routine 
activities and development project in the past, (2) to 
ease the preparation of performance based budgeting and to 
explain relationship between output/outcome, (3) to provide 
objective and proportional descriptions about government 
financial activities, and (4) to increase government 
financial statistical credibility.i6i The central government 
expenditures are divided into eleven "function" categories: 
(1) public service, (2) defense, (3) Safety (4) economy (5) 
environmental, (6) housing and public facilities, (7) 
healthcare, (8) tourism and culture (9) religion, (10) 
education, and (11) social protection .162 

The unified budget principle in this budget reform is 
very important in ensuring that investment and recurrent 
operational expenditures simultaneously are considered in 
budget decision making. It will also assist in preparing the 


161 Indonesian Ministry of Finance, State Budget Plan FY2005, 
(Jakarta: Indonesian Ministry of Finance, 2005), 162. 

162 ibid. 
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performance based budgeting manual that is currently being 
prepared by the government. Some basic changes on budget 
format due to the implementation of unified budget and 
budget accounts reclassification are as following: 


Table 9: Changes on Budget Format Due to the Implementation 
of Unified Budget and Budget Accounts Reclassification 


Old Format 

New Format 

> Expenditures Classification 
•Dual budgeting 

•Expenditures consisted of six 
types of expenditures 

including Development 

Expenditures 

> Expenditures Classification 
•Unified budgeting 

•Central Government 

Expenditures consist of eight 
type of expenditures 

^ Organization Classifications 

were not included in Government 
finance Notes and State Budget 
Law 

^ Organization Classifications 

are included in Government 

finance Notes and State Budget 
Law 

^ Sector Classification 

•Consisted of 20 sectors and 50 

sub sectors 

•Programs were sub 

classification of sector on 
routine and development budget 

^ Sector Classification 

•Consists of 11 functions and 
79 sub functions 

•Programs are classified based 
on departments and agencies 
and then compiled based on 
functions 

> Budget Allocation was based on 
sector, sub sector, and program 

> Budget Allocation is based on 
government departments and 

agencies 


Source: From Indonesian State Budget Plan FY2005, p. 961^3 


The detailed format changes especially on state 
expenditures based on economic classification can be 
described in the following table: 


163 ibid. 
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Table 10: Conversion State Expenditures Based on 
Expenditures Types 


Old Format 

New Format 

A. State Revenues and Donation 

A. State Revenues and 

I.Internai Revenues 

Donation 

1.Tax Revenues 

I. Internai Revenues 

2.Non Tax Revenues 

1.Tax Revenues 

IT. Foreign Aid Grant 

2.Non Tax Revenues 

B. State Expenditures 

II. Eoreign Aid Grant 

I.Centrai Government 

B.State Expenditures 

Expenditures 

I.Centrai Government 

1.Routine Expenditures 

Expenditures 

a.Saiary Expenditures 

1.Saiary Expenditures 

b.Goods Expenditures 

2.Goods Expenditures 

c.Interest Expense 

3.Capitai Expenditures 

d.Subsidies 

4.Interest Expense 

e.Other Routine 

5.Subsidies 

expenditures 

6 .Eoreign Aid 

2.Deveiopment Expenditures 

Expenditures 

7.Sociai Assistance 

II. Local Government 

8 .Other expenditures 

Expenditures 

II. Local Government 
Expenditures 

C. Primary Baiance 

D . Budget Surpius/deficit 

C.Primary Baiance 

E. Einancing (e.g. Eoreign 

D.Budget Surpius/deficit 

Loan and ioan from 

E. Einancing 

Nationai Bank) 


Source: From Indonesian State Budget Plan FY2005, p. 9tl64 


Budget classification is based on function and programs 
instead of on sector/sub-sector in the oid format. 
Function/sub-function is not the basis of budget aiiocation. 
Budget aiiocation is based on the proposed programs from 
departments and agencies and then the programs are 
ciassified by their functions and sub-functions. Hence, the 
detaii of the expenditure budget is the compiiation of the 
budget from departments and agencies. 

By comparing United States experience with its GPRA and 
the current progress of performance budgeting impiementation 


164 ibid. 
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in Indonesia several lessons learned can be noted. First, 
the GPRA is the law and has support from both President and 
Congress. The position as a law has provided strong power 
for department and agencies to implement it. Otherwise, the 
content of GPRA has appropriately provided information on 
how to implement the law, ranging from the strategic 
planning, performance measures, and linking performance to 
budget or budget performance and integration as well as the 
performance plans and reports. 

The Law No. 17.2003 does not provide detail like GPRA 
has since it only provides the several statements on the 
requirement to implement performance based budgeting. For 
the detail about performance based budgeting, currently, the 
government still discuss about this matter. To be able to 
emulate the success of GPRA, the Indonesian government has 
to provide the new law on the detail explanation and process 
of performance budgeting. By enacting it as a law which 
means it will need DPR approve, minimally it will take DPR 
attention and government can cooperate with the DPR from the 
earlier process of performance budgeting so that both 
parties agree with the content of the law and support the 
implementation. 

Second, since being enacted in 1993, the implementation 
of GPRA was initiated with several pilot projects and 
divided into two categories, the strategic planning, 
performance measures and the managerial flexibility waiver. 
The implementation was also being delayed due to several 
problems mentioned in Chapter II. Learning from those 
experiences, it would be appropriate for Indonesian 
Government not to speed up the implementation in 2006 in 
whole departments and agencies. It will be better if this 
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project is started with several pilot projects in all 
departments and agencies so that the budget efficiency and 
quick response towards any deviation and changes of the 
standards and procedures could be attained. 

Finally, the most important thing related to 
performance measurement and assessment is standards and 
benchmarking. The lack of performance standards and 
benchmarking available for the public sector has been a 
problem for budget audit and evaluation stage in comparing 
between performance plans and actual realization. The method 
used in GPRA of having annual performance plans and annual 
performance report to be compared in performance evaluation 
stages can be emulated. This will also save department and 
agencies time in preparing the budget instead of paying much 
attention in designing and finding appropriate standards and 
benchmarking. 

D. PBB MODEL FOR DOD 

1. Overview 

It is an ironic that Indonesia, a country with a vast 
geographical area, a high degree of vulnerability to both 
internal and external threats and a strategic location in 
the Asia-Pacific region currently has not more than 50% 
military readiness capabilities on main weaponry system. For 
example, from 200 jet fighters, comprising F-16 Fighting 
Falcons, A4-Sky Hawks, Hawk 200s and F-5s and about 75 
percent of the fighters, however, are unable to fly due to 
lack of spare parts. Several jet fighters are already 
obsolete and in need of replacement. Some A-4 Sky Hawks, for 
example, were made in 1980 but they are still being flown 
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today, roughly 13 years after they should have been grounded 
in 1990. 

It is absolutely understandable that Indonesia needs to 
acquire more sophisticated weapons and use them in defending 
its territorial integrity and protecting the country 
adequately. This condition has been exacerbated by having 
the military embargo imposed by the United States following 
the violence in East Timor in 1999. The embargo has 
prevented Indonesian National Armed Force (TNI) from getting 
spare parts, ammunition, and other military hardware from 
United States. Albeit, several talks between Indonesia and 
US have showed a promising result, however, officially U.S. 
still has not lifted the embargo. 

This has made it difficult to provide funding for the 
real needs of TNI. Defense structure is very expensive so 
that it should be planned carefully with sufficient 
considerations towards the availability and sustainability 
of state funds as well as internal and external political 
environment. The last effort by government in procuring 
four Sukhoi's through counter trade system with Russia was 
one the efforts to enhance the capabilities by using off- 
budget payment and exchanging forty commodities for several 
deliveries in the future. 

The condition above coupled with an increasing public 
attention to enhance the capabilities from increasing 
external threats as well as TNI's willingness to reform 
itself into a new paradigm; there has been a dramatic 
budget increase for the last three years on defense budget. 
For example, from only approximately US$800 million 
(FY2002$) in 2002 which was less than 4 percent of the 
government budget, for 2005, the military budget is 
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approved for approximately US$ 2.4 billion (FY2005$) which 
is almost 10% of government budget. 

The budget increase does not mean that TNI/DOD can 
spend it without having adequate control. To create 
transparency as part of civilian control over military, 
especially in budget control, all military disbursements 
should be performed in accordance with available laws and 
regulations by planning, budgeting, managing and monitoring 
them appropriately. The DPR as representative of society 
must be fully involved in the process of budgeting. A clear 
justification of confidentiality towards national security 
should be formulated sufficiently to avoid misusing of the 
funds in particular activities. The public has a right to 
know what really happens in the country and to obtain 
sufficient information about problems. 

To fulfill the demand in transforming into the new 
paradigm and to provide the strategic framework as well as 
to increase the accountability and present the transparency 
to the public, DOD has also issued the Defense White Paper 
which provides the strategic framework of how DOD/TNI 
perceives the threats, defines its capabilities, and 
articulates strategic policy for Indonesian defense. 
Before 2003, where the only strategic framework document 
was what was available in GBHN, DOD/DNI did not have any 
document such as National Military Strategy in the U.S. 
that was publicly available to provide the strategic 
framework of how DOD should run its business. 

The military budget system is much different from 
other civil government departments and agencies. 
Internally, in budget formulation and costing, DOD has 
already implemented the Planning, Programming, and 
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Budgeting System (PPBS). After introduction in the 1970's, 
the system has been adjusted gradually in accordance with 
the development of financial system in the country. The 
outputs of the PBBS system are coordinated into one budget 
request that will be submitted to the Ministry of Finance. 

For program and budget management, DOD/TNI has three 
basic principles: bottom up, top down, and one gate policy. 
The bottom up policy means that every unit hierarchically 
proposes the business need plans based on proposal as well 
as its explanation including the detail calculations and 
arguments to support the budget plan. The top down policy 
means top management determines the lower level budget 
based on priority, corporate policies, and resources 
capabilities. One gate policy means every function, 
planning, programming, executing, and controlling in 
organization is conducted separately by function in the 
organization. 

In its response to the implementation the unified 
budget, DOD has also changed its budget classification into 
programs for the FY 2005. The new Defense Budget program 
structure with functional approach can be seen as following: 
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Table 11: Indonesian DOD Budget FY 2005 Based on Functions 


(in thousands. Rupiah) 


Code 

Program 

Amount 

02.01 

Country Defense 


02.01.01 

Integrated Defense Development 

2,150,010,040 

02.01.02 

Land Defense Development 

9,052,604,211 

02.01.03 

Sea Defense Development 

3,187,952,828 

02.01.04 

Air Defense Development 

2,377,112,942 

02.01.05 

Strengthening Nation Unity and Sovereignty 

25,759,920 

02.02 

Defense Supports 


02.02.01 

Defense System and Strategy Development 

52,587,019 

02.02.02 

Defense Supporting Potency Development 

5,016,172,184 

02.02.03 

Defense Industries Development 

19,314,956 

02.03 

Overseas Military Cooperation 


02.03.01 

International Military Cooperation 

40,789,490 

02.04 

Defense Research and Development 


02.04.01 

Defense Research and Development 

28,756,549 

02.04.02 

National Defense Development 

15,500,000 

02.05 

Others Defense 


02.05.01 

TNI Social Operation 

26,569,085 

Total 


21,977,629,824 


Source: From Indonesian State Budget Plan FY2005, p.lSS^^S 


Based on the table above, although the budget has been 
adjusted to program structure, from the program title, it 
can be seen that the programs are too general. If the 
DOD/TNI is willing to use this program classification in the 
performance based budgeting, it will result in the programs 
that are too wide, consisting of many overlapping 
performance measures. Moreover it will make the process of 
setting and evaluating the performance measures difficult. 
It will be better if the program is created more 
specifically so that the performance measures can be 
directly tied to the program. 

The above configuration will be different if we compare 
them with the United States DOD Programs directed to 
specific strategies: such as Air Combat Program, Air Force 
Aircraft Operations, Air Force Depot Maintenance, Airlift 

165 ibid. 
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Program, Army Land Forces Operations, Basic Research, 
Chemical Demilitarization, Communications Infrastructure, 
Defense Health, Depot Maintenance - Naval Aviation, Depot 
Maintenance - Ship, DOD Applied Research Program, DOD Small 
Business Innovation Research Technology, Energy Conservation 
Improvement, Housing, Military Force Management, Missile 
Defense, Navy Ship Operations, Navy/Marine Corps Air 
Operations, Recruiting, Shipbuilding, and Facilities 
Sustainment, Restoration, and Modernization. 

For strategic planning, because DOD and Indonesian 
National Armed Forces (TNI) are separated and have equal 
position under the President, there is no such a unitary 
strategic planning. Ideally and officially, both 
organizations have to have the same mission, visions, goals 
and objectives, and strategies. Albeit, financially and 
administratively, TNI is willing to merge with the DOD, 
hierarchically, TNI is still reluctant to be under the DOD 
just like the U.S. has in its DOD. If this condition would 
be resolved, conflicting of interests in civil military 
relationships will continue to escalate in the future 
years. The current TNI mission, visions, and corporate 
level goals are: 


108 



Table 12: Indonesian TNI Mission, Vision, Goals 


Mission Statement 

"To build a solid institution of Indonesian Armed 
Forces (TNI) and to give priority to the 
professionalism as the guardian of the nation 
freedom and safety." 

Vision Statement 

"To complete the internal reformation and put TNI 
back as the nation tool of defense by holding on to 
Propenas 2002 - 2004 decision of defense and safety 
development." 

Corporate Level 
Goals 

1. Restructure the TNI in accordance with new 

paradigm consistently through reposition, 

redefinition , and reactualization of TNI's role 
to protect, maintain and defend the unitary state 
of the Republic of Indonesia against internal and 
external threats by incorporating the human 
rights and perform services in assisting the 
development of the country 

2. Develop the nation defense and security system 
capabilities 

3. Enhance the quality of TNI professionalism, 
enhance the force ratio of main components, and 
develop regional capabilities supported with 
appropriate infrastructure and funding. 

4. Widen and enhance the quality of defense 

bilateral cooperation to maintain the stability 
of regional safety and to participate in 

maintaining the world peace. 

Strategies 

1. Build the Force Structure: Build military 

operational capabilities to defend and not for 
defense, enhancing management quality, 

intelligent capabilities, territorial functions 
enhancement, and cooperation with allies. 

2. Force development through limited personnel and 
equipment regeneration and personnel management 
with task orientation. 

3. Enhancing the quality Main Weaponry System 

through Research and Development activities and 
existing main weaponry system maintenance 

4. Military Deployment is developed in conjunction 

with defense policies and strategies with 

geographical configuration orientation. 


2. The Cascade Model of Performance Based Budgeting 
for Indonesian DOD/TNI 

Based on the comparative study with United States 
experiences with GPRA in Chapter II as well as the 
theoretical literature review in Chapter III, the framework 
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of the cascade model of Performance Based Budgeting for 
Indonesian DOD/TNI can be described in following figure: 


Figure 2: Performance Based Budgeting Process 
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The PBB process is started by designing the strategic 
plan. Strategic planning consists of mission, goals and 
objectives, and strategies that provide strategic direction 
for the organization and inspire the entire human resources 
in the DOD/TNI to better manage and conduct all necessary 
activities to achieve the mission. To be able to design an 
effective performance measurement system, DOD/TNI has to 
prepare the strategic plan so that it can generate a sound 
measurement system. 


no 


















The strategic plan has to be well documented. The 
example of a strategic plan can be described in the 
following figure: 

Figure 3: Example of DOD/TNI Strategic Plan 

DOD/TNI Strategic Plan 
(FY 2006-2011) 

Mission Statement: To build a solid institution of Indonesian 

Armed Forces (TNI) and to give priority to the 
professionalism as the guardian of the nation 
freedom and safety. 

Vision Statement: To complete the internal reformation and put 

TNI back as the nation tool of defense by holding 
on to Propenas 2002 - 2004 decision of defense and 
safety development. 

Strategic Goal 1: Restructure the TNI in accordance with new 

paradigm consistently through reposition, 
redefinition, and reactualization of INI's role 

Strategic Goal 2: Develop and maintain the nation defense and 

security system capabilities 

□ Strategic Objective 2.1. Fulfill the optimum existing force 
readiness level 

• Strategies 2.1.1.1. Enhancing the quality Main Weaponry 
System through Research and Development activities 

• Strategies 2.1.1.2. Enhancing the quality Main Weaponry 
System through maintaining existing main weaponry system 
configuration 

□ Strategic Objective 2.2. Fulfill TNI Personnel capabilities 
up to 95 percent 

Strategic Goal 3: Enhance the quality of TNI professionalism, 

enhance the force ratio of main components, and 
develop regional capabilities supported with 
appropriate infrastructure and funding. 

Strategic Goal 4: Build and enhance the quality of defense 

bilateral cooperation to maintain the stability of 
regional safety and to participate in maintaining 
the world peace. 


From the figure above, it can be seen that the 
strategic goal is more specifically defined by two 
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strategic objectives, one of which is "Fulfill the optimum 
main weaponry system readiness level." The strategic 
performance goal is a measurable target level to be 
achieved by FY 2011 in which it is the measurable outcome 
that DOD/TNI wants to achieve. To achieve this objective, 
two strategies are provided: (1) Enhancing the quality Main 
Weaponry System through Research and Development activities 
and (2) Enhancing the quality Main Weaponry System through 
maintaining existing main weaponry system configuration. 

In linking strategic goals and objectives with 
performance measures, in Eigure 3 below, the goal is 
divided in two types: Strategic Performance Goals and 
Annual Performance Goals. A strategic performance goal will 
function as a long term measurement in medium term 
framework while an annual performance will function as an 
intermediate outcome for specific fiscal year budget. To 
achieve the 80% mission capable rates of existing main 
weaponry system by 2011, for EY 2006, the intermediate 
outcome is assumed at 60% from 2005 baseline in the 
percentage of aircraft that are capable of performing their 
designated mission. 

Based on the strategies in Eigure 2, a series of 
programs have to be created to achieve the goals and 
objectives. In the figure 3, the example of program is Air 
force Depot Maintenance Program. The program will have 
three performance measures, an output and two outcomes. The 
output is the number of the number of maintenance actions 
performed in EY 2006 and completed on schedule, 1250 
actions in which this the number of maintenance actions 
that will be covered in EY2006 budget. The two outcomes for 
the programs are first "Reduce the percentage of 
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maintenance actions performance in the depot that are 
completed on schedule" and second "Maintain no more than 
30% Organic Aircraft Quality Defect Rate (defects per 
aircraft)." 

Figure 4: Example of End Outcome and Intermediate Outcome 

to outputs 


strategic Goal 2: Develop and maintain the defense and 

security system capabilities 

□ Strategic Objective 2.1. Fulfill the optimum main weaponry 
system readiness level 

^ Strategic Performance Goal 2.3.1. Maintaining 80% 
mission capable rates of existing main weaponry system 
by 2011 

O FY 2006 Annual Performance Goal 2.3.1.1. Achieve 60% 
from 2005 baseline in the percentage of aircraft that 
are capable of performing their designated mission. 

D Air force Depot Maintenance Program 

• Program Measure 2.3.1.1 - PM.A. Output: The 
number of maintenance actions performed in FY 
2006 and completed on schedule is 1250. 

• Program Measure 2.3.1.1 - PM.B. Outcome: Reduce 
the percentage of maintenance actions 
performance in the depot that are completed on 
schedule 

• Program Measure 2.3.1.2 - PM.C. Outcome: 

Maintain no more than 30% Organic Aircraft 
Quality Defect Rate (defects per aircraft) 


In budget preparation stage, two processes have to be 
performed: budget formulation and costing in DOD/TNI and 
budget plan evaluation in the Ministry of Finance. The 
output of budget formulation and costing is such an initial 
budget request. Deriving strategic planning and performance 
measures in previous stages, in budget formulation and 
costing, DOD/TNI have to provide the scope, content. 
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performance and quality of programs and activities proposed 
to meet the organization's mission, goals and objectives. 

In this stage, the most important tool that has to be 
applied is selecting the costing method. DOD/TNI has to 
choose which method would be appropriate for their 
business, between full cost and ABC. Probably most 
government departments and agencies would use the full cost 
method since it is easier to align the budget account from 
previous format, however since the DOD/TNI has used unit 
cost in designing the budget, ABC might be the good choice 
for the costing method. 

In preparing a budget request that will be delivered 
to the Ministry of Finance, the budget request includes a 
strategic plan which provides a logical connection between 
the strategic and performance context and day to day basis 
activities, performance information in terms of output, 
intermediate outcome, final outcome, efficiency, and 
effectiveness, and cost information which identifies the 
real cost of providing services with cost charged to the 
appropriate program. The information on the budget request 
above can be used to provide budget justification that will 
be negotiated in the Ministry of Finance, to make tradeoffs 
among the subunits in allocating the funds based on 
availability and priorities; to measure productivity of the 
subunits, and to determine overlapping services within the 
DOD/TNI. In the Ministries of Finance, a performance 
informed budget will be used in analyzing the budget 
request from TNI/DOD in the context of its performance 
implications or outcomes, to make tradeoffs among the 
departments and agencies, and to provide budget 
justification for submission to the DPR. 
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The budget approval stage involves two players, 
government represented by the Ministry of Finance as well 
as DOD/TNI and the DPR. In this stage, to obtain approval 
from the DPR and to defend the budget request, performance 
information can help the government's capacity to assess 
competing claims in the budget by arming the DPR with 
better information on the performance results for both 
individual program as well as entire government budget 
portfolios in addressing performance outcomes. Furthermore, 
it helps clarify the relationship between the budget and 
the results, in which the performance information not only 
shows the results of a certain budget program but also 
indicates how those funds are expected to generate those 
results. To maximize control, the DPR could focus primarily 
on expected results rather than in the amount of funds 
appropriated. This would assist agencies in developing 
priorities, since authorization of legislation involves 
reaching consensus between the DPR and the Government. 

In the budget execution stage, the important point is 
how to continuously update and gather of performance 
information. The information should be reported at 
frequent, specified dates, with department heads being held 
responsible for both accurate data and improvements in 
performance. The potential benefits for DOD/TNI of using 
performance information in this stage are to understand the 
specific implications of the approved budget for 
performance, to analyze the progress of the programs by 
comparing the targeted performance with the current 
progress, to assist in making contract decision and its 
management as well as measuring the contractor performance, 
and to monitor the budget and performance realization and 
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provide quick feedback for unintended and intended 
deviation budget execution. 

For budget audit and evaluation, the performance 
information can be used to shift the focus of audits and 
evaluations to include performance questions, rather than 
only financial compliance so that internal and external 
auditors are able to determine the success or failure of a 
program as well as the compliance with applicable law. 
Generally, the performance information will help the work 
of auditor in conducting performance audits. Without having 
a performance system, auditors are not able to conduct the 
audits because in performance audit, the organization's 
performance measurement system has to be compared with 
standards and benchmarks. Specifically, the performance 
measure elements such resources, inputs, outputs, outcomes 
would be used to assess the economy (resources VS inputs), 
efficiency (inputs VS outputs), and cost effectiveness 
(resources VS outcomes) and program effectiveness (outputs 
VS outcomes) aspects of programs or organizations. 

E. CHAPTER SUMMARY 

For the last few years, Indonesia has proved itself to 
be a more democratic country by incorporating a series of 
reform to enhance government's accountability. In the 
budgeting system, the State Finance Law No. 23 2003 provides 
significant changes on the state, regional and local 
budgeting system through four important changes: performance 
based budgeting, unified budget. Medium Term Expenditure 
Framework, and Government Accounting Standard. Progress such 
as budget account reclassification by implementing the 
unified budget as well as strategic planning can be found in 
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regional and local 


almost every department, agency, 
government and has proved that the government has speeded up 
the PBB implementation. 

In the DOD/TNI, as an agency which obtains significant 
budget increase for the last 5 years, the performance based 
budgeting has to be well provided so that public can be 
better informed on how DOD manages the budget. The 
performance based budgeting model provided in this Chapter 
can be used as an initial step to implement the PBB. 

The PBB model starts with the strategic planning and 
consists of mission, goals and objectives, and strategies 
that provide strategic direction for the organization and 
inspire the entire human resources in the DOD/TNI to better 
manage and conduct all necessary activities to achieve the 
mission. The performance measures that will be used are 
output, outcome, economy, efficiency, and effectiveness. In 
linking the performance to budget, the budget request 
including the strategic performance context, performance 
information, and cost information has to be applied and 
provided in the four stages of budgeting, budget 
preparation, budget approval, budget execution, and budget 
evaluation and audit. 


117 



THIS PAGE INTENTIONALLY LEET BLANK 


118 



V. 


CONCLUSIONS AND RECOMMENDATIONS 


A. CONCLUSIONS 

Performance based budgeting has been acknowledged as 
the new proven successful method budget reform in the 
developed countries such as United States, New Zealand, 
Sweden and United Kingdom. The willingness of Indonesian's 
government to adopt performance based budgeting would 
provide an umbrella for government performance evaluation 
tool in enhancing public sector accountability. However 
this effort should result from the inner will to change 
instead of to fulfill demand and pressure from the donor 
countries. 

To permeate a performance perspective into budget 
information and decision, the underlying performance 
information should be credible, sufficient, and used by all 
budget decision makers. The government has to involve the 
stakeholders in preparing the PBB implementation, 
especially, the DPR in which both will get involved in 
preparing the rules and regulations so that this effort 
would not be a hollow paperwork exercise. 

More importantly, to emulate the success of GPRA, the 
government and DPR has to enact a new law for the blueprint 
or detailed content of PBB to be applied in government 
departments and agencies so that they would be challenged 
and strongly encouraged to appropriately take the PBB 
effort as a priority. The role of the legislature (DPR) in 
cooperating with government is very critical. Not only 
because DPR has to get involved for the preparation of the 
law as mentioned above but also because it is DPR that 
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would significantly employ the PBB in the process of budget 
approval and evaluation. 

The decision from the government to fully implement PBB 
for the FY2006 perhaps should be intensely evaluated. 
Currently, the biggest challenge to implement the PBB is 
lack of understanding of the concept of strategic planning 
and performance measures; even many public managers do not 
understand performance measures such as the difference 
between output and outcome. Selecting several pilot projects 
in each department or agency might be more effective so that 
any deviation and suggestion for improvement can be quickly 
realized. 

The opportunities by having the Law No. 25/2004 on 
National Development Planning System have to be exploited to 
better result in the departments and agencies strategic 
planning. Instruction for the preparation has to be well 
designed and clearly enunciated so that departments and 
agencies can provide strategic framework that actually can 
be linked to day to day activities and inspire all the 
elements in the organization to shift to the new paradigm 
and new way doing of business. The instruction is very 
important because it is not possible that they would capture 
the old long term planning frameworks from GBHN. In this 
matter, the Medium Term Expenditure Framework (MTEF) aimed 
to enhance the inter-correlation between planning and 
budgeting process and to obtain more rational and strategic 
resources allocation. 

In selecting the performance measures that would be 
applied, the measures have to be adjusted to the need of the 
country as a developing country. Specifically, although 
efficiency and effectiveness have been dominantly used in 
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the developed countries where Indonesian government can 
obtain the benchmarking, the perspective of economy must be 
considered as the first priority. As a developing country 
which has limited resources to be allocated, the aspect of 
economy to better obtain the lowest price among alternatives 
as well as cost reduction on current expenditures would 
provide cost saving and optimum resource allocation. 
Additionally, due to the lack of performance standards and 
benchmarking available for the public sector, the method 
used in GPRA by having annual performance plans and annual 
performance report to be compared in performance evaluation 
stages can be emulated. 

Especially in DOD/TNI, the economy perspective is very 
critical due to many procurements funded by Overseas Export 
Credit with its high lobbying cost, and short grace loan 
period, and high interest rates, and the new approach from 
Indonesian's government in developing national military 
industries to create and develop main weaponry system. The 
more cost reduction obtained from the lower price on 
procurement, the more optimum budget allocation can be 
obtained, and finally the more weapons can be procured by 
using the same amount budget. 

The current system cost per unit measures such as cost 
per flying hour, cost per steaming hour, cost per tank mile 
that have been used in preparing the current budget might 
be considerations in selecting the costing method for the 
performance based budgeting between the full cost method 
and the ABC method. Additional work to be done is to 
establish a blueprint that provides manual instruction of 
the ABC and as basis for the creating a performance based 
costing. 
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Attention also must be focused on the strategic 
planning for DOD and TNI that must contain the same 
mission, vision, goals and objectives, and strategies as 
the inspirational framework in doing the business and 
developing the defense sector as well as the national 


military 

industries. 

It 

is time for 

both 

sides 

to sit 

together. 

get rid 

of 

the sectoral 

ego. 

and 

think 

strategically on how 

to 

build a solid 

institution 

and to 


give priority to the professionalism as the guardian of the 
nation's freedom and safety as well as to complete the 
internal reformation in the defense sector. 

B. RECOMMENDATIONS FOR FUTURE RESEARCH 

This study provides an overview of performance based 
budgeting and a basic model for Indonesian DOD/TNI. The 
study's intent is to promulgate the Indonesian DOD/TNI to 
better prepare the further requirements of the performance 
based budgeting process that would be implemented in 2006 . 
Within the study performance based budgeting in Indonesian 
defense sector, there are several areas to conduct 
supporting research as following: 

• How specifically the performance based 
budgeting would assist the stakeholders 
decision making in allocating the limited 
resources ? 

• How performance could specifically be measured 
and differed in terms of war deployment and 
peacetime operation? 

• Within Indonesian DOD/TNI how specifically the 
performance would be measured as well as how 
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the performance measurement system would be 
appropriately designed? 

• How the Activity Based Costing (ABC) would be 
used as a costing method in DOD/TNI? 

• How the performance based budgeting concept 
would be linked to the business activities 
within Indonesian DOD/TNI such as in 
acquisition (performance based acquisition) and 
in logistic (performance based logistic) . 
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APPENDIX A. PROGRAM ASSESSMENT RATING TOOL TYPES OF 

PROGRAMS 


(Source: From 0MB PART Instruction FY 2006) 
TYPES OF PROGRAMS 


Although most PART questions are the same, the PART divides 
all programs into seven categories for the purpose of 
asking additional questions unique to a particular type of 
program. These categories apply to both discretionary and 
mandatory programs. 

1. Direct Federal Programs where services are provided 
primarily by employees of the Federal government, such as 
the National Weather Service and the Visa and Consular 
Services. 

2. Competitive Grant Programs that provide funds to State, 
local and tribal governments, organizations, individuals 
and other entities through a competitive process, such as 
Health Centers. 

3. Block/Formula Grant Programs that provide funds to State, 
local and tribal governments and other entities by 
formula or block grant, such as Weatherization Assistance 
and the Ryan White program. 

4. Regulatory Based Programs that accomplish their mission 
through rulemaking that implements, interprets or 
prescribes law or policy, or describes procedure or 
practice requirements, such as the Food Safety and 
Inspection Service. 

5. Capital Assets and Service 

Acquisition Programs that achieve their goals through 
development and acquisition of capital assets (such as 
land, structures, equipment, and intellectual property) 
or the purchase of services (such as maintenance, and 
information technology) , for example. Defense 
Shipbuilding and the Bonneville Power Administration. 
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6. Credit Programs that provide support through loans, loan 

guarantees and direct credit, such as Export-Import 

Bank/Long Term Guarantees. 

7. Research and Development 

(R&D) Programs that focus on knowledge creation or its 
application to the creation of systems, methods, 
materials, or technologies, such as the Department of 
Energy/Solar Energy and NASA/Mars Exploration programs. 


There is a separate PART form for each of the seven types 
of Federal programs, though most of the questions are 
common across the seven forms. The vast majority of Federal 
programs fit into one of the seven categories of programs 
for which there is a PART. However, some programs use more 
than one mechanism to achieve their goals (e.g., grants and 
credit) . Even in these cases, using one PART is likely to 
be sufficient. To enable this for R&D programs, which can 
use one of the other program types (e.g., competitive 
grants) as a means of funding R&D, the R&D PART has been 
designed to enable R&D programs that fund research through 
grants, contracts, cooperative agreements or other 
transactions to answer questions from the Competitive 
Grants instrument. Similarly, R&D programs that construct 
or operate equipment or facilities will answer some 
questions from the Capital Assets and Service Acquisition 
PART. There may be other cases in which drawing questions 
from two different PARTS - i.e., creation of a "mixed" form 
- yields a more informative assessment. In those instances, 
the PART type that most closely reflects the core functions 
of the program should be chosen as a base, and then, if 
necessary, selected questions from another PART can be 
added. The 0MB examiner should consult with a member of the 
0MB Performance Evaluation Team, if considering this 
approach. 

For new programs for which it is impractical to expect 
results, it is possible to complete only Sections 1 through 
III of the PART. However, performance measures, targets and 
related information should still be provided (and scored) 
in Section IV for new programs where practical. 

Question-specific instructions are attached to help explain 
the purpose of each question and general standards for 
evaluation. The individual PART worksheets also contain 
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this guidance as well as instructions on the technical 
aspects of using the worksheets. These instructions will 
not cover every case, and it is up to the user to bring 
relevant information to bear in answering each question 
that will contribute to the program's assessment. 


133 



THIS PAGE INTENTIONALLY LEET BLANK 


134 



APPENDIX B. PROGRAM ASSESSMENT RATING TOOL 

QUESTIONS 


(Source: 0MB PART Instruction FY 2006) 

I. PROGRAM PURPOSE & DESIGN 

This section examines the clarity of program purpose and 
soundness of program design. It looks at factors including 
those the program, agency, or Administration may not 
directly control but which are within their influence, such 
as legislation and market factors. Programs should 
generally be designed to address a either an efficiency 
matter, such as a public good or externality, or a 
distributional objective, such as assisting low-income 
families in the least costly or most efficient manner. A 
clear understanding of program purpose is essential to 
setting program goals, measures, and targets; maintaining 
focus; and managing the program. Potential source documents 
and evidence for answering questions in this section 
include authorizing legislation, agency strategic plans, 
performance plans/performance budgets, and other agency 
reports. 

Options for answers are Yes, No or Not Applicable. Design 
flaws in the underlying legislation can and should be 
considered and supported by evidence, and are grounds for a 
No. Not Applicable answers are likely to be rare, 
particularly for items 1.1-1.4, as these questions should 
apply to virtually all programs. (For R&D programs, most of 
the questions in this section help address program 
"relevance," one of the three fundamental issues of the R&D 
Investment Criteria (see Appendix A).) 

1.1 Is the program purpose clear? 

Purpose of the question: to determine whether the program 
has a focused and well-defined mission. Determining this 
purpose is critical to determination of useful 
performance measures and targets. 

Elements of a Yes answer : A Yes answer would require a 
clear and unambiguous mission. Considerations can include 
whether the program purpose can be stated succinctly. A 
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No answer would be appropriate if the program has 
multiple conflicting purposes. 

Evidence/Data : Evidence can include a statement of the 
purpose and supporting objectives from the program's 
authorizing legislation, program documentation or mission 
statement. 

1.2 Does the program address a specific and existing 
problem, interest, or need? 

Purpose of the question: to determine whether the program 
addresses a specific problem, interest, or need that can 
be clearly defined and that currently exists. 

Elements of a Yes answer : A Yes answer would require the 
existence of a relevant and clearly defined interest, 
problem or need that the program is designed to address. 
A Yes answer would also require that the program purpose 
is still relevant to current conditions (i.e., that the 
problem the program was created to address still exists). 
Considerations could include, for example, whether the 
program addresses a specific market failure. A No should 
be given if there is no clear need for the program. 

Programs may receive a Yes to question 1.1 and a No on 
question 1.2 and vice versa. 

Eor example. Housing and Urban Development's HOPE VI 
program had a clear purpose with a goal of demolishing 
100,000 of the most severely distressed public housing 
units. However, the program has surpassed this goal and 
addressed the need for which the program was originally 
created. Therefore, the program received a Yes in 1.1 and 
a No in 1.2. In addition, the Department of Education 
Vocational Education program had an unclear mission 
caused by multiple and overlapping objectives. It was 
able to document, however, that a significant number of 
students are graduating from high school and community 
college without the necessary academic and technical 
skills. Therefore, the program received a No for 1.1 and 
a Yes for 1.2. 

Evidence/Data: Evidence should describe the problem, 
interest or need that the program is designed to address 
and include relevant documentation. An example could be 
the number and income levels of uninsured individuals for 
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a program that provides care to those without health 
insurance. 

1.3 Is the program designed so that it is not redundant or 
duplicative of any other Federal, State, local or private 
effort ? 

Purpose of the question : to determine whether the program 
is designed to fill a unique role or whether it instead 
unnecessarily duplicates or even competes with other 
Federal or non-federal programs. 

Elements of a Yes answer : A Yes answer would require that 
the program does not excessively overlap with other 
Federal or non-federal efforts, including the efforts of 
State and local governments or the private and non-profit 
sectors. A consideration can include whether the program 
serves a population not served by other programs. 

A No answer should be given when there is more than one 
program that addresses the same problem, interest, or 
need, regardless of the size or history of the respective 
programs. For programs that partially overlap with 
others, a No should be given when major aspects of the 
program, such as its purpose, targeted beneficiaries, or 
mechanisms, are duplicative. If there are two programs 
that significantly overlap and one is large and another 
is small, both programs should receive a No for this 
question. 

Similar programs might be justified in receiving a Yes if 
a strong case can be made that fixed costs are low and 
competition is beneficial (e.g., perhaps multiple 
laboratories) or if more than one service delivery 
mechanism is appropriate (e.g., block grants for base 
activities and competitive grants for demonstration 
projects). Also, the standard of evidence to receive a 
Yes should be higher where Federal programs overlap with 
each other than where a Federal program overlaps with 
private, local, or State programs. For example, two 
Federal programs to address training would face a high 
standard to receive a Yes to this question; in contrast, 
a Yes could be provided to a Federal program to address 
indigent medical care across the nation, even though 
there are many local and private programs that also 
address indigent medical care. The key would be whether 
the gaps in the non-Federal provision are large enough to 
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warrant a Federal program and whether the Federal program 
is well designed to mesh with non-Federal efforts and 
responsibilities. 

For credit programs, a Yes answer would require evidence 
of the market failure/absence or unwillingness of 
private-sector participation and an overview of the 
market, including all international. Federal, local, and 
private-sector participants. 

For R&D programs, some degree of duplication is 
permissible, if it is well justified and coordinated. A 
Yes answer would require justification that the program 
provides value beyond that of any similar efforts at the 
agency, efforts at other agencies, or efforts funded by 
State and local government, private and non-profit 
sectors, or other counties. Justification first requires 
due diligence in identifying similar past or present 
efforts . 

Evidence/Data : Evidence should identify duplicative 
programs and their total expenditures and/or a 
description of efforts supported by those programs that 
address a similar problem in a similar way as the program 
being evaluated. 

1.4. Is the program design free of major flaws that would 

limit the program's effectiveness or efficiency? 

Purpose of the question : to determine whether there are 
major design flaws in the program that limit its 
efficiency. 

Elements of a Yes answer : The program should be free from 
major design flaws that prevent it from meeting its 
defined objectives and performance goals. To receive a 
Yes, there should be no strong evidence that another 
approach or mechanism would be more efficient or 
effective to achieve the intended purpose. A 
consideration could be whether the government would get 
the same or better outcome by expending fewer total 
resources through a different mechanism. Eor example, 
there may be evidence that a regulatory program to ensure 
public safety would be more effective than a grant 
program. Analysis should consider whether the program 
structure continues to make sense given changing 
conditions in the field (e.g., changing threat levels or 
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social conditions) . Other considerations could include 
whether the program extends its impact by leveraging 
funds and contributions from other parties. 

For credit programs, an additional consideration can 
include the extent to which a large number of borrowers 
would otherwise not have access to financial resources. 
Also consider whether the program costs are adequate, but 
not excessive, to achieve the policy goals. For example, 
a Yes answer could mean that the program effectively uses 
market mechanisms to reduce government risk and thus 
minimize program costs. 

Regulatory programs should receive a No if the statute 
underlying the regulations is not designed to maximize 
net benefits. 

For capital assets and service acquisition programs, a 
Yes answer requires that, in addition to the general 
criteria, the program is supported by an adequate capital 
asset management infrastructure that is consistent with 
the principles and techniques of effective capital 
programming, and has clear lines of authority, 
responsibility, and accountability for managing capital 
assets. 

Evidence/Data : Evidence demonstrating efficient design 
can include cost effectiveness studies comparing 
alternative mechanisms (e.g., regulations or grants) with 
the current design (say, direct federal provision). 
Evidence on the relative benefits and costs of the 
activity are also useful. 

Evidence for determining whether the threshold for 
capital programming has been met should include the 
documented program-relevant agency or bureau capital 
programming policies, directives, instructions, manuals, 
and assignment of authorities and responsibilities to 
agency personnel and organizational units. 

1.5 Is the program design effectively targeted, so that 
resources will reach intended beneficiaries and/or 
otherwise address the program's purpose directly? 

Purpose of the question : to determine whether the program 
is designed so that program resources will reach the 
intended beneficiaries efficiently and to avoid 


139 



unintended subsidies. "Beneficiaries" refers to those who 
benefit from the favorable outcome of the program. 
"Reach" refers to the distribution of benefits. 

Unlike Question 1.4, which addresses examination of 
alternatives to achieve a program's goals, this question 
asks whether program resources under the chosen 
alternative are oriented toward the effective achievement 
of the program's purpose. 

Elements of a Yes answer: A Yes answer would require that 
a program demonstrates that the right beneficiaries are 
being targeted, activities that would have occurred 
without the program are not subsidized (or receive only 
warranted levels of subsidies) , and program funds are 
targeted effectively to meet program purposes. 

Acceleration of activities due to Federal funding can be 
grounds for a Yes, but there should be evidence that the 
acceleration warrants the subsidy or application of 
funding. Acceleration of an activity that increases 
profits for a business -- that the firm would or could 
have undertaken eventually without the subsidy or 
application of funding -- would not generally qualify for 
a Yes, unless there are significant external (i.e., 
social) benefits from the activity. 

In the case of block-grant and credit programs, the 
assessment should also consider how well funds are 
targeted to meet the program purpose and whether funds 
are protected against supplantation or substitution. 

For R&D programs, a Yes answer would require 
identification of relevance to specific national needs, 
agency missions, fields of science or technology, or 
other "customer" needs. A customer may be another program 
at the same or another agency, an interagency initiative 

or partnership, a firm_^^_or an organization from another 

sector or country. For these programs, the question 
refers to awardees of contracts, cooperative agreements 
or other transactions, as well as grants. 

Evidence/Data: Evidence should show that the program is 
designed to 1) reach the highest practicable percentage 
of target beneficiaries, and 2) have the smallest 
practicable share of funds going to unintended 
beneficiaries. Regarding item 1, a small program may only 
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be able to reach a small number of beneficiaries, but it 
should be well targeted on some merit basis. On item 2, 
programs not designed to avoid unwarranted shares of 
funding going to beneficiaries who do not need or merit 
the funding should receive a No answer. Programs that are 
designed in a way that is likely to result in significant 
levels of erroneous payments should receive a No. 


II. STRATEGIC PLANNING 

This section focuses on program planning, priority setting, 
and resource allocation. Key elements include an assessment 
of whether the program has a limited number of performance 
measures with ambitious -- yet achievable -- targets, to 
ensure planning, management, and budgeting are strategic 
and focused. Potential source documents and evidence for 
answering questions include strategic planning documents, 
agency performance plans/performance budgets and reports, 
reports and submissions from program partners, evaluation 
plans, and other program documents. 

Options for answers are Yes, No or Not Applicable. While it 
is recognized that some programs may have greater 
difficulty than others in developing quantitative 
performance goals, programs must have meaningful and 
appropriate methods for demonstrating results. 0MB and 
agencies should work together to develop approaches for 
programs where it is difficult to develop quantitative 
measures, and where qualitative, expert-review, or other 
measures are more appropriate. For R&D programs, most 
questions in this section help address the prospective 
aspects of the R&D Investment Criteria (see Appendix A). 

2.1 Does the program have a limited number of specific 
long-term performance measures that focus on outcomes and 
meaningfully reflect the purpose of the program? 

Purpose of the question : to determine if the program has 
long-term performance measures to guide program 
management and budgeting and promote results and 
accountability. This question seeks to assess whether the 
program measures are salient, meaningful, and capture the 
most important aspects of program purpose and appropriate 
strategic goals. 
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Elements of a Yes answer : A Yes requires two or three 
specific, easily understood outcome measures that 
directly and meaningfully support the program's purpose. 
"Long-term" means a long period relative to the nature of 
the program, likely 5-10 years, and consistent with time 
periods for strategic goals used in the agency's GPRA 
strategic plan. 

The performance measures should focus mainly on outcomes, 
although in some cases output measures are permissible. 
The measures may be those developed to comply with GPRA, 
so long as they meet the "Performance Measures" section 
of this document. Otherwise, 0MB and agencies should 
revise the measures to fully meet GPRA and PART 
standards. A Yes answer can also be given if 0MB and the 
agency have reached agreement on a limited number of 
long-term measures that will be added to the 2006 GPRA 
strategic plan or performance plan/performance budget. 
Significant changes to the GPRA strategic plan may 
require stakeholder consultation. 

Output measures only meet the standards of a Yes answer 
if the program can produce sound justification for not 
adopting outcome measures. For example, a program that is 
exclusively focused on processing applications and is 
unable to adequately define a quantifiable outcome 
measure may use measures that focus on increases in 
accuracy and/or timeliness of service delivery. Whenever 
output measures are proposed, the program must clearly 
show how such measures reflect progress toward desired 
outcomes. 

An example of an unacceptable long-term measure is a 
housing program that is proposing using a measure of 
"number of housing units constructed." In this case, such 
an output is unacceptable as the program is expected to 
articulate and measure progress toward achieving outcomes 
(e.g., increases in homeownership rates, increases in 
housing equity in low-income communities). 

For more detailed discussion on when output measures may 
be used as proxy measures for outcomes, please see 
"Selecting Performance Measures" section of the PART 
guidance or visit 0MB's website at 
http://WWW.omb.gov/part/ . 
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A No must be given for long-term measures that do not 
directly and meaningfully relate to the program's purpose 
or are unnecessarily focused on outputs and lack adequate 
justification. A program must not receive a No for having 
too many measures, if it has identified a few high- 
priority ones that represent important aspects of the 
program. 

Performance measures should be listed in the Measures tab 
of the PART worksheet. Only measures that meet the 
standards for a Yes should be entered on the worksheet. 

For block grant programs that support a wide range of 
purposes and allow grantees to set their own program 
priorities, measures that address the extent to which 
grantees meet their own goals or effectively target 
populations are options if no better measures are 
possible. 

For R&D programs, 0MB will work with agencies to assess 
existing or develop appropriate measures. Some R&D 
programs, especially in basic research, may not be 
required to provide long-term efficiency measures, due to 
the uncertainty of outcomes and the years it takes to 
achieve and recognize them. 

Evidence/Data : Evidence will include the long-term 
measures established for the program either in the 
existing agency GPRA documents or other program documents 
or as agreed to by 0MB and to be included in the 2006 
GPRA documents. In the case of new measures, if targets 
and baselines are not defined, a plan for their 
development (i.e., timeline, methods for data collection, 
responsible office and/or staff) must be agreed to by the 
agency and 0MB. 

2.2 Does the program have ambitious targets and timeframes 

for its long-term measures? 

Purpose of the question : to determine if the program has 
challenging but realistic quantifiable targets and 
timeframes for the long-term measures. 

Elements of a Yes answer: A Yes answer requires that 
specific quantified targets have been developed for 
measures evaluated in Question 2.1. (Where targets are 
not "quantitative," they still must be verifiable, e.g.. 
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through a clean audit or outstanding ratings by an expert 
panel). Baselines from which to measure targets are 
encouraged for outcome measures and required for output 
measures. Targets and timeframes must be ambitious, that 
is, they must be set at a level that promotes continued 
improvement within achievable efficiencies. Where 
relevant, a Yes also requires that a program has defined 
an appropriate end target. 

A No is appropriate if quantified targets or timeframes 
are not included for key measures or if the targets or 
timeframes are not ambitious or challenging. 

If the program received a No in Question 2.1, the program 
must get a No for this question. 

Targets must be listed in the Measures tab of the PART 
worksheet. 

For R&D programs, a Yes answer would require that the 
program provides multi-year R&D objectives. Where 
applicable, programs must provide schedules with annual 
milestones, highlighting any changes from previous 
schedules. Program proposals must define what outcomes 
would represent a minimally effective program and a 
successful program. 

Evidence/Data: Evidence will include targets in existing 
agency GPRA documents or other program documents or as 
agreed to by 0MB and to be included in the 2006 GPRA 
documents. 

2.3 Does the program have a limited number of specific 
annual performance measures that can demonstrate progress 
toward achieving the program's long-term goals? 

Purpose of the question: to determine whether a limited 
number of annual performance measures have been 
identified that directly support the long-term goals 
evaluated in Questions 2.1 and 2.2. The measures should 
be logically linked to the long-term goals in a manner 
that enables them to demonstrate progress toward 
achieving those long-term goals. 

Elements of a Yes answer: A Yes answer would require 
annual performance measures that are discrete, 
quantifiable, and measurable. Most importantly, these 
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annual measures should measure the program's progress 
toward reaching the long-term goals evaluated in 
Questions 2.1 and 2.2. The annual performance measures 
may focus on outputs and may or may not be those 
developed by the agency to comply with GPRA. A Yes answer 
can also be given, if 0MB and the agency have reached 
agreement on a limited number of annual measures that 
will be included in the 2006 GPRA performance 
plan/performance budget. 

To receive a Yes answer, programs must have at least one 
efficiency measure or be developing one. If a measure is 
under development, a schedule for adoption (i.e., 
timeline, methods for data collection, responsible office 
and/or staff) must be agreed to by the agency and 0MB. 

For more detailed discussion on defining acceptable 
efficiency measures please see "Selecting Performance 
Measure" section of the PART guidance or visit OMB's 
website at http://www.omb.gov/part/ . 

Programs that are unable to provide an efficiency answer 
may only receive a Yes to this question if they are able 
to adequately explain why adoption of such a measure is 
not feasible (e.g., some basic R&D programs). 

If the program received a No in Question 2.1, an 
explanation of how annual performance goals contribute to 
desired long-term outcomes and purpose of the program 
must be provided to receive a Yes for this question. 
Performance measures should be listed in the Measures tab 
of the PART worksheet. Only measures that meet the 
standards for a Yes should be entered on the worksheet. 

For block grant programs that support a wide range of 
purposes and allow grantees to set their own program 
priorities, measures that address the extent to which 
grantees meet their own goals or effectively target 
populations are options if no other measures are 
possible. 

For R&D programs, a Yes answer would require that the 
program has annual performance measures to track how the 
program could improve scientific understanding and its 
application. For R&D programs that have multi-year 
schedules, the annual measure should tie into the longer 
term milestones, as appropriate. Some basic research 
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programs may not be able to define meaningful annual 
outcome or efficiency measures. In such cases, these 
programs may use process-related indicators (see Question 
3.4), if the program can explain how those processes are 
directed toward the intended long-term goals. 0MB will 
work with agencies to address appropriate measures. 

For capital assets and service acquisition programs, a 
Yes answer requires that, in addition to the general 
criteria, annual performance measures include those that 
are sufficient to track achievement of the cost, 
schedule, and performance goals of asset acquisitions as 
they relate to the overall program. 

Evidence/Data: Evidence will include the annual measures 
established for the program in the agency GPRA 
performance plan/performance budget or other program 
documents, or they may be new measures as agreed to by 
0MB and which will be included in the 2006 GPRA 
performance plan/performance budget. Evidence for capital 
asset acquisition programs includes agency acquisition 
and project management working documents, contract 
performance measures and metrics, and business cases (0MB 
Circular A-11 Exhibit 300s) . 

2.4. Does the program have baselines and ambitious targets 

for its annual measures? 

Purpose of the question: to determine if the program has 
baselines and challenging but realistic quantified 
targets for the annual measures. 

Elements of a Yes answer: A Yes answer would generally 
require that baselines have been established for most of 
the annual measures evaluated in Question 2.3. A Yes 
answer also requires that specific quantified annual 
targets have been developed for most of the program's 
annual measures evaluated in Question 2.3. These targets 
provide a specific value with which performance can be 
compared. These targets must be ambitious, that is they 
must be set at levels that ensure continued improvement 
and realization of efficiencies. They also should be 
within reason for the program to achieve. 

A No answer would be appropriate if quantified targets or 
timeframes are not included for most measures or if the 
targets are not ambitious or challenging. 
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If the program received a No in Question 2.3, the program 
must get a No for this question. Targets should be listed 
in the Measures tab of the PART worksheet. 

Evidence/Data: Evidence will include targets in the 
agency GPRA performance plan/performance budget or other 
program documents or as agreed to by 0MB and will be 
included in the 2006 GPRA performance plan/performance 
budget. 

2.5. Do all partners (including grantees, sub-grantees, 
contractors, cost-sharing partners, and other government 
partners) commit to and work toward the annual and/or long¬ 
term goals of the program? 

Purpose of the question: to determine whether program 
efforts carried out by program partners also support the 
annual and long-term performance goals of the program. 

Elements of a Yes answer: A Yes answer would require that 
program managers strive to ensure that partners support 
the overall goals of the program and measure and report 
on their performance as it relates to accomplishing those 
goals. Eor example, a program that requires all grant 
agreements and contracts to include performance measures 
that will help the program achieve its goals and monitor 
those measures would receive a Yes. 

If, however, a program does not through a performance 
requirement or some other means get program grantees to 
link their activities to the program's goals, a No would 
be appropriate. The most obvious example of a partner is 
an entity receiving program funding. While a program 
cannot always control the activities of its partners, it 
can exert influence through a number of various 
mechanisms. 

If the program received a No for both Questions 2.1 and 
2.3, the program must receive a No for this question. 

In the case of regulatory programs, all regulated 
entities are not necessarily defined as program partners. 

Eor programs funding grants for basic research, a Yes 
answer can be achieved if the program solicitation 
explicitly includes the program goals, and grant 
applications and progress reports provide sufficient 


147 



means for the program manager to assess performance and 
continuing relevance. 

Evidence/Data: Evidence can include contracts and other 
documents that tie contractor performance to program 
goals, as well as other procedures the program uses to 
get partners to commit to, measure, and report on 
performance related to the program's goals. 

2.6. Are independent evaluations of sufficient scope and 
quality conducted on a regular basis or as needed to 
support program improvements and evaluate effectiveness and 
relevance to the problem, interest, or need? 

Purpose of the question: to ensure that the program (or 
agency) conducts non-biased evaluations on a regular or 
as-needed basis to fill gaps in performance information. 
These evaluations should be of sufficient scope to 
improve planning with respect to the effectiveness of the 
program. (For R&D programs, this question is central to 
prospective planning to address all of the R&D investment 
criteria (see Appendix A).) 

Elements of a Yes answer: A Yes answer would require 
regularly scheduled objective, high quality, independent 
evaluations that examine how well the program is 
accomplishing its mission and meeting its long-term 
goals. A Yes answer also would require that program 
evaluations address the following elements of quality, 
scope, and independence. 

Not Applicable is not an option for this question; given 
the flexibility in determining what constitutes an 
evaluation, all programs should undergo an evaluation 
that meets the following elements of quality, scope, and 
independence. Quality. Evaluations should be sufficiently 
rigorous to provide information on the effectiveness of 
the program and, for programs that support or employ a 
range of services and approaches, information on the 
effectiveness of the various services and approaches. To 
receive a Yes, agencies should provide evidence that they 
have chosen and applied evaluation methods that provide 
the most rigorous evidence of a program's effectiveness 
that is appropriate and feasible. A program may receive a 
Yes for this question if the agency and 0MB determine 
that existing evaluations do not provide the most 
rigorous evidence possible, but that the program is in 
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the process of developing new evaluation approaches that 
will provide the most rigorous evidence possible by a 
specified future date. 

The purpose of the question is to find out whether 
agencies have evaluations that assess the effectiveness 
of the program. The most significant aspect of program 
effectiveness is impact—the outcome of the program, which 
otherwise would not have occurred without the program 
intervention. Where it is feasible to measure the impact 
of the program, randomized controlled trials are 
generally the highest quality, unbiased evaluation to 
demonstrate the actual impact of the program. However, 
these studies are not suitable or feasible for every 
program, and a variety of evaluation methods may need to 
be considered because Federal programs vary so 
dramatically. Other types of evaluations, including well- 
designed quasi-experimental studies, may provide useful 
information about the impact of a program (but should be 
scrutinized given the increased possibility of an 
erroneous conclusion) and/or can help address how or why 
a program is effective (or ineffective) (i.e., meeting 
performance targets, achieving efficiency, fulfilling 
stated purpose) . 

Overall, evaluations must be appropriate to the type of 
program. Agencies and 0MB should consult evaluation 
experts, either in-house and/or external, as appropriate, 
when deciding what type of evaluation will provide the 
most rigorous evidence appropriate and feasible. The 
following are several links to references on program 
evaluation located on the internet that are not intended 
to be exhaustive, but which may be helpful: 

Program Evaluation Methods: Measurement and Attribution 
of Program Results; Treasury Board of Canada, 
Secretariat; 1998. (a book available online) 
http://WWW.tbs-sct.gc.ca/eval/pubs/meth/pem-mep_e.pdf 

Understanding Impact Evaluation; The World Bank Group, 
(a web site) 
http://WWW.worldbank.org/poverty/impact/index.htm 

"Program Evaluation: An Evaluation Culture and 
Collaborative Partnerships Build Agency Capacity;" GAO- 
03-454; U.S. General Accounting Office; May 2003. 
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http://WWW.gao.gov/docdblite/summary.php?recflag=&accno=A 
0 67 97 &rptno=GAO-0 3-4 54 

"Performance Measurement and Evaluation: Definitions 
and Relationships;" GAO/GGD-98-26; U.S. General 
Accounting Office; April 1998. 
http://WWW.gao.gov/docdblite/summary.php?recflag=&accno=l 
60204&rptno=GGD-98-26 

"Designing Evaluations;" GAO/PEMD-10.1.4; U.S. General 
Accounting Office; May 1991. 
http://161.203.16.4/t2pbat7/144040.pdf 

Randomized Controlled Trials: A User's Guide; Jadad, 
Alejandro A.; BMJ Books; 1998. (a book available online) 
http://WWW.bmjpg.com/ret/contents.html 

Research Methods Knowledge Base; Trochim, William M.; 
Cornell University. (a web site) 
http://trochim.human.Cornell.edu/kb/ 

"Identifying and Implementing Educational Practices 
Supported By Rigorous Evidence: A User Eriendly Guide;" 
U.S. Department of Education; December 2003. 
http://WWW.ed.gov/rschstat/research/pubs/rigorousevid/rig 
orousevid.pdf 

Scope. This question looks directly at whether there are 
evaluations on the program's achievement of performance 
targets, and that these evaluations examine the 
underlying cause and effect relationship between the 
program and the target. In cases where a comprehensive 
evaluation is unnecessary based on the known 
effectiveness of an intervention and performance data on 
the program, evaluations that fill gaps in performance 
information can meet the elements of a Yes answer. A 
program's effectiveness, including impact, also may be 
considered. A program may receive a Yes for this question 
if the agency and examiner determine that existing 
evaluations are inadequate, but the program is in the 
process of developing an appropriate evaluation to be 
completed by a specified future date. 

Evaluations also should include recommendations on how to 
improve the program's performance. To ensure the program 
continues to meet its performance targets, an evaluation 
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should be scheduled on a periodic basis, such as every 
two to five years, or whatever time schedule is 
reasonable based on the specific program and its mission 
and goals. 

R&D programs also should undergo independent reviews of 
relevance to their agencies, fields of science or 
technology, or customers, (e.g., of process) in addition 
to assessing questions of performance. These reviews 
should conclude with reports documenting the findings and 
recommendations. A "customer" may be another program at 
the same or another agency, an interagency initiative or 
partnership, or a firm, an organization from another 
sector or country, or the general public. Industry¬ 
relevant programs may use industry cost-sharing of 
associated projects as an indicator of market-relevance, 
and they should incorporate industry in planning and 
prioritization. Reviews should be rigorous and methodical 
and be a critique of the program's methods, results and 
findings by others in the field with requisite training, 
expertise, and independence. 

Independence. To be independent, non-biased parties with 
no conflict of interest would conduct the evaluation. 
Evaluations conducted by the program itself should 
generally not be considered "independent;" however, if 
the agency or program has contracted out the evaluation 
to a third party this may qualify as being sufficiently 
independent. Evaluations conducted by an agency's 
Inspector General or program-evaluation office might also 
be considered "independent." 0MB examiners and agency 
staff will determine if a specific evaluation can be 
considered "independent" for this question. 

If a program has had previous evaluations that meet the 
elements of independence and scope, but that were not 
sufficiently rigorous, and the program is developing or 
about to conduct a new program evaluation using the most 
rigorous method that is feasible and appropriate, then 
the program would receive a Yes. 

In the absence of ANY independent evaluations, a program 
would receive a No, unless it is planning to carry out a 
rigorous program evaluation in the near future. A No 
answer would also be appropriate for a program that has 
insufficient independent evaluation data or has 
evaluations that address process and not performance. 
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Evidence/Data: Evidence should include a program 
evaluation plan or schedule of program evaluations and 
program documentation describing the type of evaluation, 
including scope and quality, and the criteria for 
selecting an independent evaluator. 

2.7. Are Budget requests explicitly tied to accomplishment 
of the annual and long-term performance goals, and are the 
resource needs presented in a complete and transparent 
manner in the program's budget? 

Purpose of the question: to establish whether the 
performance-planning and budget-planning processes are 
integrated so that 1) resource allocation decisions 
reflect desired performance levels (given resource 
constraints) and 2) the effects of funding and other 
policy changes on results are clear. 

Elements of a Yes answer: A Yes answer should reflect 
effective program budgeting based on sound levels for 1) 
annual and long-term performance targets and 2) budget 
resources. To receive a Yes, the program must provide a 
presentation that makes clear the impact of funding, 
policy, or legislative decisions on expected performance 
and explains why the requested performance/resource mix 
is appropriate. A program with budget planning that is 
not tied to performance or strategic planning would 
receive a No. 

A Yes answer would also require that the program report 
all direct and indirect costs needed to attain the 
performance results, including applicable agency 
overhead, retirement, and other costs that might be 
budgeted elsewhere. The exclusion of minor amounts of 
services provided from central departmental offices 
(e.g.. Office of the Secretary) from program costs does 
not require a program to receive a No answer. However, a 
program that generates significant costs - which might 
range from radioactive waste disposal to attorneys' 
salaries - that must be addressed by another program 
should budget for these costs or, at a minimum, provide 
this information in clear display tables that display the 
full costs of attaining results. 

Eor capital assets and services acquisition programs, a 
Yes answer requires that, in addition to the general 
criteria, programs explain the relationship of asset 
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acquisitions to overall program performance goals and 
would be able to identify impacts of changes on program 
performance (for example, the effect of a change in the 
quantity acquired). 

Evidence/Data: Evidence can include documentation of how 
the budget request directly supports achieving 
performance targets. Budget documents should also clearly 
indicate the full costs of achieving performance goals, 
even if some of these costs do not appear in the specific 
account or activity line of the program. 

Also, evidence can include an agency program budget 
estimate that identifies all spending categories in 
sufficient detail to demonstrate that all relevant costs 
are included or a report that shows the allocation of all 
significant program overhead costs to the program. 

2.8. Has the program taken meaningful steps to correct its 
strategic planning deficiencies? 

Purpose of the question: to determine whether the program 
is on track to correct any strategic planning 
deficiencies that have been identified. 

Elements of a Yes answer: A Yes answer would require that 
the program has acted to correct strategic planning 
deficiencies. A program that does not review planning 
efforts or does not make corrections to eliminate 
identified deficiencies would receive a No. The question 
addresses any deficiencies identified in this section. 
Particular emphasis, however, should be placed on whether 
the program is working to adopt a limited number of 
specific, ambitious long-term performance goals and a 
limited number of annual performance goals that 
demonstrate progress toward achieving the long-term 
goals, if they do not already have these measures or 
associated baselines, targets, and timeframes. 

Eor Capital Assets and Service Acquisition programs (and 
relevant R&D programs), one strategic planning deficiency 
to be addressed is if the program has in the past 
received a No to Question 4.CA1. 

Evidence/Data: Evidence can include a description of how 
deficiencies in the strategic planning of a program are 
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identified and corrected, as well as examples of such 
changes. 


Specific Strategic Planning Questions by Program Type 


Regulatory Based Programs 

2.RG1. Are all regulations issued by the program/agency 
necessary to meet the stated goals of the program, and do 
all regulations clearly indicate how the rules contribute 
to achievement of the goals? 

Purpose of the question: to determine whether the 
program had developed regulations with clearly 
specified goal (s) . It should be determined whether (1) 
the program is only issuing those rules absolutely 
necessary to achieve long-term program goals and is 
not over-regulating, (2) all of the rules necessary to 
meet the program goals have been issued, and (3) the 
regulations clearly indicate how they help to meet the 
program goals. 

Elements of a Yes answer: A Yes answer would require 
that only those regulations that are absolutely 
necessary to accomplish the program mission and goals 
are promulgated or are in the process of being 
promulgated; this determination should be based on the 
standards laid out in Executive Order 12866 on the 
identification of the need for the regulation and the 
identification of the market failure (if applicable). 
Additionally, the public should be able to understand 
how the regulations fit into the overall achievement 
of the program goals. A Yes response indicates that 
there are no superfluous regulations, that regulations 
are planned or in the process of being promulgated to 
cover regulatory gaps where new regulations are 
required to accomplish program goals, and that the 
Preamble of each program regulation indicates how the 
rule contributes to the achievement of specific 
program goals. 

A program would receive a No if it has 1) obvious 
regulatory gaps or outdated regulations in effect and 
2) not initiated planned actions to rectify these 
problems in a timely manner. 
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Evidence/Data: Evidence can include legislation that 
indicates specifically or generically what regulations 
need to be promulgated as well as the rules 
themselves, especially the preambles. It can also 
include internal agency guidance indicating a process 
is in place to ensure rulemaking involves a clear 
linkage of the need for the rule to a stated goal. In 
this case, the agency should be able to clearly 
articulate this association. 

NOTE: Questions 2.RG1, 3.RG3, and 4.RG1 address the 
progression of reviews for a program's rules. 
Specifically, 2.RG1 addresses targeted development of 
regulations, 

3.RG3 addresses effective implementation of regulations, 
and 4.RG1 addresses maximization of potential benefits 
during the regulation's implementation. 


Capital Assets and Service Acquisition Programs 


R&D programs addressing the acquisition, construction 
or operation of facilities or other capital assets 
should answer the Capital Assets and Service 
Acquisition question 2.CA1). 

2.CA1. Has the agency/program conducted a recent, 
meaningful, credible analysis of alternatives that 
includes trade-offs between cost, schedule, risk, and 
performance goals and used the results to guide the 
resulting activity? 

Purpose of the question: to determine whether the 
agency is investing in an asset or service that 
provides the best value to the government. 

Elements of a Yes answer: To receive a Yes response, 
the agency should have conducted analyses of 
alternatives and use those analyses. Each analysis 
should include the baseline assessment/status quo, 
non-material solutions (e.g., data compression in lieu 
of a new data cable), consideration of alternatives 
(including alternatives to capital assets and benefit- 
cost analysis in accordance with 0MB Circular A-94), 
and trade-offs between cost, schedule, and performance 
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goals. The program should be able to demonstrate that 
the analysis is credible (e.g., by having it reviewed 
and validated by an independent entity outside the 
program). If an independent entity's analysis differs 
from the program's analysis, the program should defend 
differences. 

Evidence/Data: Evidence can include a summary of the 
analysis of alternatives, and documentation of any 
independent reviews of the analysis. As one source of 
data, an agency capital asset plan or business case 
(Exhibit 300) documentation may also be used. 


Research and Development Programs 

R&D programs addressing the acquisition, construction 
or operation of facilities or other capital assets 
should answer the Capital Assets and Service 
Acquisition question (2.CA1). 


2.RD1. If applicable, does the program assess and compare 
the potential benefits of efforts within the program and 
(if relevant) to other efforts in other programs that 
have similar goals? 

Purpose of the question: to determine whether 
applicable programs are evaluating their efforts with 
respect to their relative potential benefits. 
Applicable programs include those applied R&D programs 
that pursue multiple options toward achieving similar 
public benefits. (This question addresses the first of 
the industry-related R&D criteria, regarding the 
articulation of program benefits (see Appendix A).) 

Elements of a Yes answer: To receive a Yes rating, the 
program (or agency) should conduct periodic 
comparisons of the potential benefits of its proposals 
with alternatives. The program should be able to 
demonstrate that the analysis is credible (e.g., by 
having it reviewed and validated by an independent 
entity outside the program). If an independent 
entity's analysis differs from the program's analysis, 
the program should defend differences. 
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Evidence/Data: Evidence can include an analysis 
comparing proposed approaches with alternative 
strategies. 0MB will work with agencies as needed to 
assist in the content and structure of these 
assessments. 


2.RD2. Does the program use a prioritization process to 
guide budget requests and funding decisions? 

Purpose of the question: to determine whether the 
program has clear priorities and uses them in budget 
requests and funding decisions. (This question 
addresses the R&D "relevance" criterion (see Appendix 
A) . ) 

Elements of a Yes answer: A Yes answer would require a 
documented process to identify priorities and use 
them, as well as an identified set of current 
priorities among program goals, objectives, and 
activities. 

R&D programs are encouraged to work with independent 
advisory bodies to help prioritize in ways that 
benefit the larger science and technology enterprise. 

Evidence/Data: Evidence can include clear statements 
of program priorities in program documentation or 
mission statements, as well as documentation of the 
priorities identified by any qualified independent 
advisory bodies. 


III. PROGRAM MANAGEMENT 

This section focuses on a variety of elements related to 
whether the program is effectively managed to meet program 
performance goals. Key areas include financial oversight, 
evaluation of program improvements, performance data 
collection, and program manager accountability. 
Additionally, specific areas of importance for each program 
type are also explored. Potential source documents and 
evidence for answering questions in this section include 
financial statements, GAO reports, IG reports, performance 
plans, budget execution data, IT plans, and independent 
program evaluations. 
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Options for answers are Yes, No or Not Applicable. (For R&D 
programs, some of the questions in this section help 
address the prospective aspects of program "quality" and 
"performance" of the R&D Investment Criteria, in addition 
to addressing general program management issues (see 
Appendix A).) 

3.1. Does the agency regularly collect timely and credible 
performance information, including information from key 
program partners, and use it to manage the program and 
improve performance? 

Purpose of the question: to determine whether the program 
collects data on performance and the performance of its 
partners and uses the data to inform program management, 
resource decisions, and program performance. 

Elements of a Yes answer: A Yes answer would require that 
the program agency regularly collect high-quality 
performance data relating to key program goals and use 
that information to adjust program priorities, allocate 
resources, or take other appropriate management actions. 
When key program activities are carried out by other 
entities, such as grantees, agencies should also consider 
their performance as well. A Yes also requires that the 
agency has collected the baseline performance data 
necessary to set meaningful, ambitious performance 
targets. 

Program partners are other agencies or intermediaries 
responsible for carrying out different aspects of the 
program and might include partner agencies, grant 
recipients, participating financial institutions, 
regulated bodies, and contractors. Timely performance 
information is information that reflects current 
performance and is current enough to be useful in program 
management. Credible performance information is 
information that is collected through a systematic 
process with quality controls to confirm the validity of 
the data. 

For capital assets and service acquisition programs, a 
consideration is whether the program uses an earned value 
management system or similar system. 

For credit programs, consider whether the agency or 
program managers regularly collect and update loan 
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performance information to effectively and consistently 
manage the portfolios. 

Some long-term basic research programs may not be able to 
define meaningful annual outcome performance measures, 
aside from process measures. In such cases, these 
programs may use process-related measures, especially 
those that can be conceptually linked to long-term 
research goals. 

Evidence/Data: Evidence can include a description of how 
the agency uses performance information in managing the 
program, as well as illustrative examples of recent 
management actions based on performance information. 
Evidence can also include steps taken by a program to 
enact necessary improvements cited by a specific 
evaluation. 

3.2 Are federal managers and program partners (including 
grantees, sub-grantees, contractors, cost-sharing partners, 
and other government partners) held accountable for cost, 
schedule and performance results? 

Purpose of the question: to determine whether the program 
managers and partners are accountable for achieving 
program results. 

Elements of a Yes answer: A Yes answer would require that 
the program agency identify the managers who are 
responsible for achieving key program results and 
establish performance standards for those managers. When 
program partners contribute to the achievement of program 
goals, a Yes would also require those partners to achieve 
specific performance standards. 

In the case of block and formula grant programs, elements 
of a Yes are not confined to complying with the law. 
Elements of a Yes can include the presence of incentives 
for managers and program partners that would encourage 
corrections in deficient programs. Eor block-grant 
programs which support a wide range of purposes and allow 
grantees to set their own program priorities, this 
question should be interpreted as whether the grantees 
hold managers accountable and encourage corrections in 
deficient programs. 
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For capital assets and service acquisition programs, a 
consideration is whether contracts include minimum 
performance thresholds, incentives for good performance, 
or other mechanisms to increase accountability. 

Some long-term basic research programs may not be able to 
define meaningful annual outcome performance measures, 
aside from process measures. In such cases, these 
programs may use process-related measures, especially 
those that can be conceptually linked to long-term 
research goals. 

Evidence/Data: Evidence can include the use of 
performance management contracts with program managers, 
or some other mechanism for incorporating program 
performance into personnel performance evaluation 
criteria. Evidence of partners' accountability can 
include requiring grant and contract awards and renewals 
to consider past performance. 

3.3 Are funds (Eederal and partners') obligated in a timely 

manner and spent for the intended purpose? 

Purpose of the question: to determine whether funds are 
administered efficiently and obligated in accordance with 
planned schedules and spent for the intended purposes. 

Elements of a Yes answer: A Yes answer would require that 
the program funds be obligated consistently with the 
overall program plan and that a limited amount of 
unobligated funds remain at the end of the year. A Yes 
answer would also require that programs and partners 
establish schedules for obligations that properly 
correspond to the resource needs of the program plan. In 
addition, a Yes answer requires that adequate procedures 
exist for reporting actual expenditures, comparing them 
against the intended use, and taking timely and 
appropriate action to correct single audit findings when 
funds are not spent as intended. 

A program would receive a No if it had significant 
erroneous payments or was in violation of the Anti- 
Deficiency Act. 


Evidence/Data: Evidence can include periodic and year-end 
spending reports from the program and its partners. 
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Evidence on expenditures can include spending reports 
that draw intended purpose from the Congressional 
Justifications, Appropriations, and program operating 
plans and match them against actual spending. For 
grantees, evidence can include grantee audit reports 
under the Single Audit Act, including data captured in 
the Federal Audit Clearinghouse, and the existence of an 
established procedure for reviewing actual expenditures 
against budgets in grant awards or appropriate Federal 
guidelines. 

A No answer is strong evidence that a No may be warranted 
for Question 3.6, which examines financial management 
more generally. 

3.4. Does the program have procedures (e.g., competitive 
sourcing/cost comparisons, IT improvements, appropriate 
incentives) to measure and achieve efficiencies and cost 
effectiveness in program execution? 

Purpose of the question: to determine whether the program 
has effective management procedures in place to ensure 
the most efficient use of each dollar spent on program 
execution. 

Elements of a Yes answer: The answer to this question 
should focus on whether the program has regular 
procedures in place to measure and achieve efficiencies 
and cost effectiveness, recognizing that various types of 
procedures may be acceptable. A Yes would require that 
the program's performance plans include efficiency 
measures and targets, such as per-unit cost of outputs, 
timing targets, and other indicators of efficient and 
productive processes germane to the program. In addition, 
there are other acceptable ways to demonstrate that a 
program has established procedures for measuring and 
achieving efficiencies and cost effectiveness. For 
example, a program that regularly uses competitive 
sourcing to determine the best value for the taxpayer, 
invests in IT with clear goals of improving efficiency, 
etc., could receive a Yes. A de-layered management 
structure that empowers front line managers and that has 
undergone competitive sourcing (if necessary) would also 
contribute to a Yes answer. For mandatory programs, a Yes 
could require the program to seek policies (e.g., through 
review of proposals from States) that would reduce unit 
costs. Also consider if, where possible, there is cross- 
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program and inter-agency coordination on IT issues to 
avoid redundancies. 

For R&D programs, efficiency measures can include program 
overhead costs, average times to fund competitive awards, 
etc. 

Evidence/Data: Evidence can include efficiency measures, 
competitive-sourcing plans, and IT improvement plans 
designed to produce tangible productivity and efficiency 
gains, or IT business cases that document how particular 
projects improve efficiency. 

3.5. Does the program collaborate and coordinate 

effectively with related programs? 

Purpose of the question: to determine whether a federal 
program collaborates with other related program(s) in a 
meaningful way. 

Elements of a Yes answer: A Yes answer would require that 
the program collaborate with related federal programs 
and, to the extent appropriate or possible, with related 
State, local, and private programs. A Yes answer would 
require that the program show evidence of collaboration 
leading to meaningful actions in management and resource 
allocation, for example, the existence of a coordinating 
council would not by itself constitute meaningful 
collaboration. (This question applies to programs that 
have interrelated, but separately budgeted, efforts. An 
example of an interrelated federal program is the shared 
effort of the Department of Veterans Affairs and the 
Medicare Program to provide care for aging veterans.) 
Meetings, discussions groups, and task forces are not 
sufficient for a Yes. A Yes requires evidence of 
collaboration leading to meaningful actions in management 
and resource allocation. 

Evidence/Data: Evidence of meaningful collaboration could 
include joint grant announcements, planning documents, 
performance goals, or referral systems. 

3.6. Does the program use strong financial management 

practices ? 
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Purpose of the question: to determine whether the program 
uses effective financial management practices in 
administering program funds. 

Elements of a Yes answer: A Yes answer would require 
that, at a minimum, the program be free of material 
internal control weaknesses reported by auditors. In 
addition, depending on the particular risks inherent to 
the program, a Yes may require meeting some or all of the 
following criteria: 

• The program has procedures in place to ensure that 
payments are made properly for the intended purpose to 
minimize erroneous payments. 

• Financial management systems meet statutory 

requirements. 

• Financial information is accurate and timely. 

• Integrated financial and performance systems support 
day-to-day operations. 

• Financial statements receive a clean audit opinion 
and have no material internal control weaknesses. 

If an agency-wide material weakness exists that is 
unrelated to the program, then a Yes response would be 
appropriate. However, if an agency-wide material weakness 
has a direct relation to the program (e.g., a lack of 
systems that support day-to-day operations), then the 
program would receive a No for this question. 

For block-grant programs which support a wide range of 
purposes and allow grantees to set their own program 
priorities, a Yes answer should also require that 
grantees (e.g.. States and localities) meet the criteria 
for effective financial management identified above. 

For credit programs, a Yes answer would also require that 
the program consistently meets the requirements of the 
Federal Credit Reform Act of 1990, the Debt Collection 
Improvement Act and applicable guidance under 0MB 
Circular A-129. 

Evidence/Data: Evidence can include recent audit reports 
and existence of procedures to identify the above-listed 
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criteria, such as the ability to measure improper 
payments. 

Unlike Question 3.3, the presence of significant 
erroneous payments does not automatically generate a No 
for this question. Specifically, if an agency has 
instituted strong financial management controls that are 
demonstrating measurable improvements in erroneous 
payments over time, then a Yes may be appropriate. 
Nevertheless, a No on Question 3.3 is strong evidence 
that a No may be appropriate for this question. 

For credit programs, evidence can include actual reports 
detailing the performance of the agency's portfolio 
management, subsidy calculations, re-estimates, 
modifications, etc. Other evidence can include 
independent evaluations of the program's performance. 

3.7. Has the program taken meaningful steps to address its 
management deficiencies? 

Purpose of the question: to determine whether the program 
has developed a system of evaluating program management 
and correcting deficiencies when they are identified. 
This question should include, but is not limited to, 
financial management or other Presidential Management 
Agenda deficiencies. However, the focus of the question 
is program-level deficiencies, as opposed to agency-level 
deficiencies that may not directly affect the program. 

Elements of a Yes answer: A Yes answer would require that 
the program has a system for identifying and correcting 
program management deficiencies and uses the system to 
make necessary corrections. A program that does not 
review program management activities and make corrections 
to eliminate identified deficiencies would receive a No. 

Evidence/Data: Evidence can include a description of how 
deficiencies in the program management are identified and 
corrected as well as examples of such changes. 


Specific Program Management Questions by Program Type 


Competitive Grant Programs 
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3. col. Are grants awarded based on a clear competitive 
process that includes a qualified assessment of merit? 

Purpose of the question: to determine whether or not 
grant funds are distributed according to a competitive 
process so that the most meritorious applications are 
awarded and so that new applicants of merit will be 
able to compete fairly with previous grant recipients. 
(For R&D competitive grants programs, this question is 
central to addressing the R&D "quality" criterion (see 
Appendix A).) 

Elements of a Yes answer: A Yes answer would require 
that the overwhelming majority of awards (95 percent 
or more of funding) are distributed according to a 
competitive process. Elements of the process can 
include independent merit review and ranking of 
applications, as well as a limit on the percentage of 
funds that are earmarked. A Yes answer would also 
require that the program operate a fair and open 
competition and provide a reasonable amount of 
outreach to encourage the participation of new 
grantees. Eair and open does not mean that data should 
not be protected (e.g., to meet classification needs, 
to protect personal data or, for an appropriate span 
of time, intellectual property.) or that other 
reasonable qualification standards can't be imposed 
(e.g. meets status as an academic organization). 
Considerations can include whether the program tends 
to provide grants to the same list of grantees year 
after year. Awards may be renewed and considered 
competitive if the original award was competitively 
awarded, renewals only extend for a short and definite 
period of time and the possibility of renewal is 
announced in the original competition.. 

Eor R&D programs, the same competitive hurdle should 
apply for external projects, although a tightly 
defined internal program can be maintained if the 
agency can clearly show that it is needed to provide 
for the maintenance of scientific expertise in-house. 
Unsolicited, potentially breakthrough ideas and out- 
of-the-box proposals can be part of the portfolio if 
they are merit-reviewed from the beginning. If such 
proposals compete fairly against a significant number 
of other proposals for a share of funding, they may be 
considered to meet the competitive standard even if 
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they don't respond to a specific request for proposal. 
In addition to grants, allowable transactions may 
include mechanisms such as contracts or other 
transactions. (Merit review processes and ranking of 
applications should also apply to these transaction 
vehicles.) 


Evidence/Data: Evidence can include a description of 
the awards process, percentage of funds earmarked, and 
percentage of funds subject to peer review. Evidence 
can also include the relative number of new awardees 
per award cycle and technical assistance and outreach 
efforts of the agency. 


3.C02. Does the program have oversight practices that 
provide sufficient knowledge of grantee activities? 

Purpose of the question: to determine whether or not 
the program has an understanding of how its funds are 
utilized by grantees. 


Elements of a Yes answer: A Yes answer would require 
that a program have sufficient oversight capacity. 
This capacity may be demonstrated by a program that 
has a reporting system in place to document grantees' 
use of funds in eligible activity categories, conducts 
site visits to a substantial number of grantees on a 
regular basis, audits grantee performance, and tracks 
actual expenditures to verify that funds are used for 
their designated purpose. A program with a strong 
relationship to its grantees and a high level of 
understanding of what grantees do with the resources 
allocated to them would receive a Yes. 


A program with a poor reporting system to track 
expenditures by grantees would receive a No rating. 


Eor R&D programs, the above standards would apply, but 
in addition to grants and grantees they would apply to 
contracts, cooperative agreements and other 
transactions and their awardees. 


Evidence/Data: Evidence can include 

structure, oversight techniques, audit 
schedule, and/or an assessment of 
quality. 


the reporting 
or site visit 
program data 
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3.cos. Does the program collect grantee performance data 
on an annual basis and make it available to the public in 
a transparent and meaningful manner? 

Purpose of the question: to determine whether or not 
the program has a system in place to collect and 
present publicly information that captures the most 
important impacts of program performance. 

Elements of a Yes answer: A Yes would require that the 
program collects, compiles and disseminates grantee 
performance information in an accessible manner, such 
as via a web site or widely available program reports. 
Data would be both aggregated on a program-wide level 
and disaggregated at the grantee level. The assessment 
about the appropriate level of aggregation of results 
may depend upon needs to protect certain data, such as 
classified data, personal data or, for a limited span 
of time, intellectual property. 

A program would receive a No if grantee performance 
information is not available to the public, or if it 
is only aggregated at a high level. Similarly, a 
program could receive a No response if the data it 
presents are not related to the impact of the program. 

For R&D programs, the above standards would apply, but 
in addition to grants and grantees would apply to 
contracts, cooperative agreements and other 
transactions and their awardees. 

Evidence/Data: Evidence can include citations of the 
types of data that are collected and disseminated as 
well as a description of how these data are made 
available. 


Block/Formula Grant Programs 

3.BF1. Does the program have oversight practices that 
provide sufficient knowledge of grantee activities? 

Purpose of the question: to determine whether or not 
the program has an understanding of how its funds are 
utilized by grantees. 
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Elements of a Yes answer: A Yes answer would require 
that a program have sufficient oversight capacity. 
This capacity may be demonstrated by a program that 
has a reporting system in place to document grantees' 
use of funds in eligible activity categories, conducts 
site visits to a substantial number of grantees on a 
regular basis, audits grantee performance, and tracks 
actual expenditures to verify that funds are used for 
their designated purpose. A program with a strong 
relationship to its grantees and a high level of 
understanding of what grantees do with the resources 
allocated to them would receive a Yes. 

A program with no reporting system to track 

expenditures by grantees would receive a No. 

Evidence/Data: Evidence can include the reporting 

structure, oversight techniques, audit or site visit 
schedule, and/or an assessment of program data 
quality. 

3.BE2. Does the program collect grantee performance data 
on an annual basis and make it available to the public in 
a transparent and meaningful manner? 

Purpose of the question: to determine whether or not 
the program has a system in place to collect and 
present publicly information that captures the most 
important impacts of program performance. 

Elements of a Yes answer: A Yes answer would require 
the program collects, compiles and disseminates 
grantee performance information in an accessible 
manner, such as via a web site or widely available 
program reports. Data would be both aggregated on a 
program-wide level and disaggregated at the grantee 
level. 

A program would receive a No if grantee performance 
information is not available to the public, or if it 
is only aggregated at a high level. Similarly, a 
program could receive a No response if the data it 
presents are not related to the impact of the program. 

Evidence/Data: Evidence can include citations of the 
types of data that are collected and disseminated as 
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well as a description of how these data are made 
available. 


Regulatory Based Programs 

3.RG1. Did the program seek and take into account the 
views of all affected parties (e.g., consumers; large and 
small businesses; State, local and tribal governments; 
beneficiaries; and the general public) when developing 
significant regulations? 

Purpose of the question: to determine the level of 
coordination, during the rulemaking process, with 
parties affected by the regulations. 

Elements of a Yes Answer: A Yes would require the 
program solicits the opinions of affected parties on 
significant regulations and thoroughly evaluates the 
concerns and suggestions raised by these entities. For 
example, a program that seeks the opinions of affected 
parties and incorporates their suggestions or explains 
why other suggestions were not incorporated during the 
rule making process could receive a Yes. 

If the program drafts its rules in a vacuum without 
consulting any of the potentially affected parties, it 
would likely receive a No. While the element of 
seeking views is mandated by law, the assessment 
should consider the extent to which the program takes 
those views into account. 

Evidence/Data: Evidence can include notices seeking 
public comment and addressing comments in final rules, 
regulation preambles which discuss compliance with the 
Regulatory Flexibility Act, Unfunded Mandates Act of 
1995, Small Business Regulatory Enforcement Fairness 
Act (SBREFA) of 1996, E.O. 13132, and National 
Environmental Policy Act, and detailed preamble 
language discussing how public comments were 
considered and addressed. 

3.RG2. Did the program prepare adequate regulatory impact 
analyses if required by Executive Order 12866, regulatory 
flexibility analyses if required by the Regulatory 
Flexibility Act and SBREFA, and cost-benefit analyses if 
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required under the Unfunded Mandates Reform Act; and did 
those analyses comply with 0MB guidelines? 

Purpose of the question: to determine whether the 
program, in justifying its rules, prepares sound 
analyses (i.e., cost-benefit analysis, risk analysis) 
that are rigorous, thorough, and based upon the best 
available data and consistent with 0MB's economic 
analysis guidelines. 

Elements of a Yes Answer: A Yes answer should 
generally include, but is not limited to, a statement 
of need of the proposed action, an examination of 
alternative approaches, and an analysis of the 
incremental benefits and costs of the proposed action. 
In accordance with 0MB's economic guidelines, 
programs' regulatory actions should maximize net 
benefits; consider regulatory and non-regulatory 
alternatives; monetize regulatory costs and benefits; 
and choose the alternative that maximizes net benefits 
in their regulatory analyses. For example, programs 
that fully document the impacts on public health and 
safety and the regulated industry through a thorough 
benefit, cost and risk analysis based upon the best 
possible available data, and examine other regulatory 
and non-regulatory alternatives would receive a Yes. A 
program may receive a Yes if its analyses are 
subjected to peer review by government entities, 
academia, industry, or non-profit research 
organizations; however, peer review is not required 
for a program to receive a Yes. 

If a program's impact analyses fail to include a 
discussion of the costs of restrictions on the 
regulated industry, a No response to this question 
would be appropriate. If the program certifies that 
regulations would not have a significant economic 
impact on a substantial number of small entities, the 
program must provide adequate justification for that 
conclusion to get a Yes answer. 

Evidence/Data: Evidence can include regulatory impact 
analyses, regulatory flexibility analyses, and cost- 
benefit analyses for the program's rules, any reports 
or feedback generated by outside reviewers, and 
coordination between reviewers and the sponsoring 
agency or program. 
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3.RG3. Does the program systematically review its current 
regulations to ensure consistency among all regulations 
in accomplishing program goals? 


Purpose of the question: to determine whether the 
agency met the goal intended when developing the 
regulation. It should be clear that the program 
consists of only those regulations that are: (1) 

necessary in achieving its goals, (2) relevant to the 
current societal and economic situation, and (3) 
complementary and consistent with each other. 

Elements of a Yes Answer: A Yes answer would require a 
program to review its regulations periodically (e.g., 
every two years) to ensure that they were consistent 
with program policies. A consideration would include 
whether the program makes attempts to minimize 
regulatory burden through constant review of 
regulations, with an eye toward streamlining, if 
possible. An additional factor to consider is whether 
the program ensures that every regulation is 
consistent with the program's goals. An example of a 
Yes could be a program that conducts look-back studies 
every third year on all of its significant regulations 
to ensure that they are all current, consistent, and 
relevant to the program goals, and, if the review 
concludes that a regulation is no longer necessary, 
the program proposes or takes action to remedy the 
situation. 

If a program, however, continues to enforce 
regulations that are no longer justified and/or 
necessary, the program would receive a No. In 
addition, a program that has not reviewed its 
regulations for consistency and necessity in a 
significant time period would receive a No. 

Evidence/Data: Evidence should include: 

• a program plan or process to conduct this exercise 
on a regular basis; 

• documentation (such as assessments, evaluations, or 
examinations; planned or completed) generated as a 
result of the above processes or plans; and/or 
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• as a result of this type of review, any changes 
made to or eliminations from the program or its 
regulations, or a justification that no change is 
needed. 

NOTE: Questions 2.RG1, 3.RG3, and 4.RG1 address the 

progression of reviews for a program's rules. 
Specifically, 2.RG1 addresses targeted development of 
regulations, 3.RG3 addresses effective implementation 
of regulations, and 4.RG1 addresses maximization of 
potential benefits during the regulation's 
implementation. 

3.RG4. Are the regulations designed to achieve program 
goals, to the extent practicable, by maximizing the net 
benefits of its regulatory activity? 

Purpose of the question: to determine whether the 
program, as it promulgates regulations, ensures that 
its regulatory requirements, in total, maximize net 
benefits. (Note that this question relates to the 
promulgation of regulations, as opposed to their 
implementation.) 

Elements of a Yes Answer: A Yes answer would require 
the program to maximize overall net benefits to the 
greatest extent practicable. The program should show 
that it makes the best effort to assess how each 
additional regulation adds to the current level of 
regulatory requirements and keeps regulatory 
compliance burden at a minimum, including the burden 
associated with information collection. Any additional 
compliance burdens should be shown to increase 
benefits substantially. An important consideration for 
this question is whether in promulgating its 
regulations, for a given level of benefits, the agency 
allows alternative methods for compliance, record 
keeping, and reporting to minimize the cost burden on 
regulated entities (including electronic means). 
Programs should be judged within their statutory 
framework; the program should maximize net benefits to 
the extent allowed by statute in order to get a Yes. 

Evidence/Data: Evidence can include statistics on 

compliance reporting burden and the costs of the 
program's requirements on regulated industries in 
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total, or evidence from a thorough cost-benefit 
analysis. 


Capital Assets and Service Acquisition Programs 

3.CA1. Is the program managed by maintaining clearly 
defined deliverables, capability/performance 
characteristics, and appropriate, credible cost and 
schedule goals? 

Purpose of the question: to determine whether the 
agency has clearly identified and defined the required 
quality, capability, and performance characteristics 
or objectives expected of the end product/result of 
the asset or service acquisition. This element is 
critical because it assures that all parties 
(government, contractor, etc) are working toward the 
same end-product and result. 

Elements of a Yes answer: If acquiring a capital 
asset, a Yes would require the program to validate 
planning decisions (procurement phase step III.l of 
the Capital Programming Guide), document the 
capabilities or characteristics that are expected, and 
to make management decisions based on whether 
milestones are being met. For example, a weapon system 
that has defined key performance parameters and 
operational requirements would get a Yes, one that is 
proceeding without such definition should receive a 
No. For services, a Yes would require the program made 
adequate use of performance-based and fixed-priced 
type contracts. A program that acquires services 
through other than performance-based and fixed-priced 
type contracts should receive a No, unless there is a 
legitimate reason for not using such contracts. 

Evidence/Data: Evidence can include program 
documentation of planning decision validation for 
asset acquisitions, and documentation describing key 
performance characteristics and/or deliverables and 
demonstration that this information is used 
appropriately in management decisions. 
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Credit Programs 


3.CR1. Is the program managed on an ongoing basis to 
assure credit quality remains sound, collections and 
disbursements are timely, and reporting requirements are 
fulfilled? 

Purpose of the question: to determine whether the 
program agency and its partners manage the financial 
performance of their credit programs. 

Elements of a Yes answer: A Yes answer would require 
managing the program based on the results of an 
effective monitoring system that tracks the financial 
performance of each credit facility. Collection and 
analysis of borrower repayment streams should be part 
of the evaluation process and could be coupled with 
reports from or trips to the field. The evaluation 
should also include an analysis of agency efforts to 

reduce default ratesT-and maximize collections and 

recoveries. 

Evidence/Data: Evidence can include quarterly 

financial statements from the program, agency. 
Treasury, the guaranteed lender, loan servicing agent; 
internal evaluations, external independent performance 
evaluations; reports from field representatives or 
trips to the field on the borrowers' performance. 

3.CR2. Do the program's credit models adequately provide 
reliable, consistent, accurate and transparent estimates 
of costs and the risk to the Government? 

Purpose of the questions: to determine whether the 
program uses a reliable method for estimating program 
costs. 

Elements of a Yes answer: A Yes answer would require 
that the agency employ a rigorous cost-estimation 
model that adequately accounts for the government's 
risk and generates dependable cost estimates for each 
cohort. A program whose cost estimates routinely 
differ markedly from actual results should receive a 
No. 

Evidence/Data: Evidence can include a description of 
any problems or advantages of the program's cost 
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estimation model. For example, a strong model could 
employ well established statistical estimation 
techniques that have a solid track record for 
predicting borrower activity. However, a weaker model 
could fail to account for potential contingencies that 
increase government risk. 


Research and Development Programs 


R&D programs addressing the acquisition, construction or 
operation of facilities or other capital assets should 
answer the Capital Assets and Service Acquisition 
question (3.CA1). 

R&D programs that use competitive grants, contracts, 
cooperative agreements or other transactions should 
answer the Competitive Grants questions (3.C01, C02, 

C03) . 

3.RD1. For R&D programs other than competitive grants 
programs, does the program allocate funds and use 
management processes that maintain program quality? 

Purpose of the question: to determine whether the 
program uses a clearly stated, defensible method for 
allocating its R&D funding. This question is central 
to addressing the R&D "quality" criterion (see 
Appendix A). 

Elements of a Yes answer: A Yes answer would require 
that the program allocate funding using a broadly 
competitive process based on merit, or that it has 
compelling justifications for R&D funding allocated 
through other means. Interpretations of competition 
and merit review should be consistent with the 
definitions in Circular A-11: "...intramural and 

extramural research programs where funded activities 
are competitively awarded following review for 
scientific and technical merit." All program funds 
allocated through means other than unlimited 
competition must document the processes they use to 
distribute funds to each type of R&D performer (e.g., 
federal laboratories, federally funded R&D centers, 
universities, etc.) . Programs are encouraged to use 
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external assessment of the methods they use to 
allocate R&D and maintain program quality. 

Evidence/Data: Evidence can include a description of 
the awards process, percentage of funds earmarked, 
percentage of funds subject to competitive peer 
review, and results of external assessments. 


IV. PROGRAM RESULTS/ACCOUNTABILITY 

This section considers whether a program is meeting its 
long-term and annual performance goals. This section also 
assesses how well the program compares to similar programs 
and how effective the program is based on independent 
evaluations. Potential source documents and evidence for 
answering questions in this section include GPRA 
performance reports, evaluations, GAO reports, IG reports 
and other agency documents. Assessments of program results 
should be based on the most recent reporting cycle or other 
relevant data. The Measures tab in the PART worksheet 
contains data fields for a performance targets and results, 
and should be completed to the greatest extent possible for 
all measures agreed to by 0MB and the agency for Section 
IT . 

Answers in this section are rated as Yes, Large Extent, 
Small Extent, and No. Like Sections I-III, the scoring 
system in this section remains on a 0 to 1 point scale. 
Scoring for this section differs by including the option of 
partial credit between 0 and 1 in increments of 0, .33, 

.67, and 1. 

In general. Not Applicable answers are not appropriate for 
Questions 4.1 and 4.2. While it is recognized that some 
programs may have great difficulty developing quantitative 
performance goals, programs are strongly encouraged to have 
some meaningful and appropriate methods for demonstrating 
results. 0MB and agencies should work together to develop 
approaches for programs where it is difficult to develop 
quantitative measures, and where qualitative, expert- 
review, or other measures are more appropriate. Not 
Applicable is a potential answer for Question 4.3 if the 
program is already operating at very high efficiency 
levels. It may also be a possible answer for Question 4.4 
and, with specific justification, for Question 4.5. (Eor 
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R&D programs, most of the questions in this section help 
address the retrospective aspects of the R&D Investment 
Criteria, with emphasis on the "performance" criterion (see 
Appendix A).) 

4.1. Has the program demonstrated adequate progress in 
achieving its long-term performance goals? 

Purpose of the question: to determine whether the program 
is meeting or making measurable progress toward meeting 

the long-term performance goals evaluated in Questions 

2.1 and 2.2. The question also seeks to determine whether 
the program's partners are meeting long-term goals 
evaluated in Question 2.5, if partner performance is 
critical to the program achieving its goals. Examples of 
partners can include grantees, participating financial 
institutions, regulated bodies, or suppliers. 

Elements of a Yes answer: A Yes answer (i.e., full 
credit) would require that the program is on track to 

meet all the long-term performance goals - including 

ambitious targets and timeframes - evaluated in Questions 
2.1 and 2.2. A program would not receive a Yes answer by 
simply meeting any one of its long-term targets, or by 
having performance measures but no ambitious targets and 
timeframes. A Yes answer would also require that, where 
applicable, partners commit to long-term outcome targets 
and achieve them as well. Where relevant, a Yes answer 
would also require that a program has addressed 
appropriately any predefined end targets. 

Partial credit, such as Large Extent or Small Extent, 
should be given in cases where there is partial, but 
notable, achievement of long-term targets. A program 
could receive a No if it had received a Yes for achieving 
its annual targets (next question), but is not making 
progress toward meeting its long-term goals. 

Additional rating guidance: 

• If adequate outcome (or output) measures are not 
available and a program received a No in Question 2.1, 
the program must receive a No answer to this question. 

• If the program received a Yes in Question 2.1 and a 
No in Question 2.2, then the program cannot receive a 
rating higher than Small Extent. 
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The only exceptions to this guidance are in cases in 
which 0MB has approved the use of alternative forms of 
assessment, as discussed in the Section IV overview. 

Evidence/Data: Evidence can include data from the 
agency's GPRA performance report, a strategic plan, or 
other Administration goals and objectives. Reports 
detailing customer satisfaction with program performance, 
program reports detailing rates of utilization or 
participation, or independent evaluations of the 
program's performance may also be considered as relevant 
evidence. In cases where targets are not met, additional 
evidence can include an explanation of the main reasons. 

Space is provided in the Measures tab of the PART 
worksheet to list and document goals, targets and 
achieved results. Only measures that meet the standards 
for a Yes should be entered on the worksheet. 

4.2. Does the program (including program partners) achieve 

its annual performance goals? 

Purpose of the question: to determine whether the program 
is meeting the targets evaluated in Question 2.4. The 
question also seeks to determine whether the program's 
partners are meeting annual targets evaluated in Question 
2.5, if partner performance is critical to the program 
achieving its overall targets. Examples of partners can 
include grantees, contractors, participating financial 
institutions, regulated bodies, or suppliers. 

Elements of a Yes answer: A Yes answer (i.e., full 
credit) would require that the program meet all the 
annual performance targets evaluated in Question 2.4. A 
Yes answer would also require the program received a Yes 
for Questions 2.1 and 2.3, and a Yes or Not Applicable 
for Question 2.5. A program would not receive a Yes 
answer by simply meeting any one of its annual targets. A 
Yes answer would also require that, where applicable, 
partners commit to annual targets and achieve them as 
well. 

Partial credit such as Large Extent or Small Extent, 
should be given in cases where there is partial, but 
notable, achievement of targets. 
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Additional rating guidance: 


• If a program received a No in Question 2.3, the 
program must receive a No answer to this question. 

• If the program received a Yes in Question 2.3 and a 
No in Question 2.4, then the program cannot receive a 
rating higher than Small Extent. 

The only exceptions to this guidance are in cases in 
which 0MB has approved the use of alternative forms of 
assessment, as discussed in the Section IV overview. 

Evidence/Data: Evidence can include data from the 

agency's annual GPRA performance report, a strategic 
plan, or other Administration goals and objectives. In 
cases where targets are not met, additional evidence can 
include an explanation of the main reasons. 

Space is provided in the Measures tab of the PART 
worksheet to list and document goals, targets and 
achieved results. Only measures that meet the standards 
for a Yes should be entered on the worksheet. 

4.3. Does the program demonstrate improved efficiencies or 
cost effectiveness in achieving program goals each year? 

Purpose of the question: to determine whether management 
practices have resulted in efficiency gains over the past 
year. 

Elements of a Yes answer: A Yes would require that the 
program demonstrate improved efficiency or cost 
effectiveness over the prior year. Efficiency 

improvements should generally be measured in terms of 
dollars or time. Programs that complete an A-76 

competition - an indicator of cost-efficient processes - 
would also likely be eligible for a Yes answer, provided 
that the competition addresses the program's key cost and 
performance drivers. Also, programs that clearly 
demonstrate very high levels of efficiency through other 
means may receive a Yes without documenting increasing 
efficiency over time. A program would normally not be 
eligible for a Yes answer to this question if it received 
a No in Question 3.4. 
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Evidence/Data: Evidence can include meeting performance 
targets to reduce per unit costs or time, meeting 
production and schedule targets; or meeting other targets 
that result in tangible productivity or efficiency gains. 
Efficiency measures may also be considered in Questions 
4.1 and 4.2. 

4.4. Does the performance of this program compare favorably 
to other programs, including government, private, etc., 
with similar purpose and goals? 

Purpose of the question: to determine how well the 
program performs relative to other programs engaged in a 
similar activity. 

Elements of a Yes answer: A Yes answer would require the 
program compare favorably to other programs with similar 
purpose and goals. Programs are not limited to Eederal 
government and can include State and local government and 
the private sector. The user should consider relevant 
evaluations that allow a comparison of programs with 
similar purpose and goals. A Not Applicable rating is 
appropriate if 1) no comparable federal, state, local 
government, or private sector programs exist, or 2) the 
comparison would be too inherently difficult and costly 
to perform for the foreseeable future. The explanation 
for Not Applicable should explain why the comparison is 
inherently too difficult to perform. (This question is 
not limited to comparisons of programs with explicitly 
coordinated "common measures.") 

Eor capital assets and service acquisition programs, 
review of performance should include cost/schedule 
adherence, quality, and quantity of deliverables. 

Evidence/Data: Evidence can include evaluations and 
documentation comparing similar programs. 

4.5. Do independent evaluations of sufficient scope and 
quality indicate that the program is effective and 
achieving results? 

Purpose of the question: to determine whether the program 
is effective based on independent and comprehensive 
evaluations. This question may be particularly important 
for programs that have substantial difficulty formulating 
quantitative performance measures. (Eor R&D programs. 
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this question is central to retrospective assessment of 
all of the R&D criteria (see Appendix A).) 

Elements of a Yes answer: A Yes answer would require that 
independent program evaluations indicate that the program 
is effective. (If the evaluations of Question 2.6 address 
past effectiveness and accomplishments, they may be used 
as evidence for this question as well.) The quality of 
evaluations of the program presented in Question 2.6 
should strongly be considered in answering this question. 
The most definitive data supporting a program's overall 
effectiveness would be from a randomized controlled 
trial, when appropriate and feasible. Data from other 
evaluation methods, such as quasi-experimental and non- 
experimental, can be considered as detailed in Question 
2.6, but should be scrutinized given the increased 
possibility of an erroneous conclusion. If a program is 
taking necessary steps to correct deficiencies uncovered 
by the evaluation, the user should address this effort in 
Question 3.7. 

Relevant evaluations would be at the national program 
level, rather than evaluations of one or more program 
partners, and would not focus only on process indicators 
such as the number of grants provided, or hits on a web 
site. Relevant evaluations would consider a program's 
impact and effectiveness. Evaluations conducted by the 
program itself should not be considered "independent." 
However, if the program has contracted out the evaluation 
to a third-party, it might be considered independent. 
Evaluations conducted by an agency's Inspector General or 
program-evaluation office also might be considered 
"independent." 0MB examiners and agency staff will 
determine if a specific evaluation can be considered 
"independent" for this question. 

Not Applicable is not an option for this question; given 
the flexibility in determining what constitutes an 
evaluation, all programs should undergo an evaluation 
that meets the elements of quality, scope, and 
independence detailed in Question 2.6. 

Evidence/Data: Evidence can include findings of an 
evaluation conducted by the General Accounting Office, 
Inspectors General, academic and research institutions, 
agency contracts, or other independent entities. 
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Specific Results Questions by Program Type 


Regulatory Based Programs 

4.RG1. Were programmatic goals (and benefits) achieved at 
the least incremental societal cost and did the program 
maximize net benefits? 

Purpose of the question: to determine whether the 
program met its goals in the most efficient way 
possible. It should be determined whether the program 
maximized net benefits through implementation of its 
regulatory actions (as opposed to regulatory 
development). In calculating the incremental costs of 
a new regulation, these costs should be compared to a 
baseline or, in a small number of cases, a less 
stringent alternative. This question deals with the 
actual implementation of the regulatory action, not 
just the conception and promulgation of the regulatory 
action. 

Elements of a Yes answer: A Yes answer would require 
that the program's implementation of the regulatory 
action maximizes net benefits. For example, a 
Department of Transportation maximum load regulation 
that demonstrates that 1) the realized benefits to 
health and safety outweigh the realized incremental 
costs of compliance, and 2) the net benefits of the 
regulatory approach adopted are higher than reasonable 
alternatives, would receive a Yes. 

If a program's regulations result in greater 
incremental costs than benefits, or if the program 
does not analyze alternative approaches to demonstrate 
that its regulatory approach maximized net benefits, 
the program should get a No. 

Evidence/Data: Evidence can include Regulatory Impact 
Analyses (RIA) or other supporting programmatic 
analyses (with supporting data after implementation 
that assumptions were correct), look-back studies, 
independent evaluations, or additional impact analyses 
using retrospective data. If a No answer is 
attributable to statutory requirements to regulate 
despite the fact that incremental costs exceed 
benefits, these statutory requirements should be 
included in the evidence section. 
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NOTE: Questions 2.RG1, 3.RG3, and 4.RG1 address the 
progression of review for a program's rules. 
Specifically, 2.RG1 addresses targeted development of 
regulations, 3.RG3 addresses effective implementation 
of regulations, and 4.RG1 addresses maximization of 
potential benefits during the regulation's 
implementation. 


Capital Assets and Service Acquisition Programs 

4.CA1. Were program goals achieved within budgeted costs 

and established schedules? 

Purpose of the question: to determine whether valid 
program goals were achieved within budgeted costs and 
established schedules and whether the program spends 
funds as planned and budgeted. For capital assets 
procured and in operation, this question also 
addresses management-in-use (i.e., the operations and 
disposal costs). 

Elements of a Yes answer: A Yes answer would require 
that the program achieved the goals evaluated in 
Section II on budget and on schedule. An example of a 
program that could receive a No rating could be an 
acquisition program that has experienced 60 percent 
cost growth and is behind schedule. If a program's 
cost and schedule targets were changed in the last 12 
months specifically due to failure to achieve previous 
goals, the program should get a No. 

Evidence/Data: Evidence can include a comparison of 
the contract schedule, deliverables, and costs with 
the final outcomes for that fiscal year. 


Research and Development Programs 

R&D programs addressing the acquisition, construction or 
operation of facilities or other capital assets should 
answer the Capital Assets and Service Acquisition 
question (4.CA1). 
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For Research and Development Programs: R&D Investment 
Criteria 

As another initiative of the President's Management 
Agenda, the development of explicit R&D investment 
criteria builds on the best of the planning and 
assessment practices that R&D program managers use to 
plan and assess their programs. The Administration has 
worked with experts and stakeholders over the past two 
years to build upon lessons learned from previous 
approaches. 

This year, the content of the R&D investment criteria has 
not changed from last year, but 0MB has worked to clarify 
its implementation. Specifically, agencies should use the 
criteria as broad guidelines that apply at all levels of 
federally funded R&D efforts, and they should use the 
PART as the instrument to periodically evaluate 
compliance with the criteria at the program level. To 
make this possible, the R&D PART has been modified to 
clarify its alignment with the R&D criteria. The R&D 
criteria are reprinted here as a guiding framework for 
addressing the R&D PART. More details will follow on 
other aspects of R&D criteria implementation this year. 

The R&D criteria address not only planning, management, 
and prospective assessment but also retrospective 
assessment. Retrospective review of whether investments 
were well-directed, efficient, and productive is 
essential for validating program design and instilling 
confidence that future investments will be wisely 
invested. Retrospective reviews should address continuing 
program relevance, quality, and successful performance to 
date. 

While the criteria are intended to apply to all types of 
R&D, the Administration is aware that predicting and 
assessing the outcomes of basic research in particular is 
never easy. Serendipitous results are often the most 
interesting and ultimately may have the most value. 
Taking risks and working toward difficult-to-attain goals 
are important aspects of good research management, and 
innovation and breakthroughs are among the results. 
However, there is no inherent conflict between these 
facts and a call for clearer information about program 
goals and performance toward achieving those goals. The 
Administration expects agencies to focus on improving the 
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management of their research programs and adopting 
effective practices, and not on predicting the 
unpredictable. 

The R&D investment criteria have several potential 
benefits: 

• Use of the criteria allows policy makers to make 
decisions about programs based on information beyond 
anecdotes, prior-year funding levels, and lobbying of 
special interests. 

• A dedicated effort to improve the process for 
budgeting, selecting, and managing R&D programs is 
helping to increase the return on taxpayer investment 
and the productivity of the federal R&D portfolio. 

• The R&D investment criteria will help communicate the 
Administration's expectations for proper program 
management. 

• The criteria and subsequent implementation guidance 
will also set standards for information to be provided 
in program plans and budget justifications. 

• The processes and collected information promoted 
under the criteria will improve public understanding 
of the possible benefits and effectiveness of the 
federal investment in R&D. 

The R&D Investment Criteria 


The Relevance, Quality, and Performance criteria apply to 
all R&D programs. Industry- or market-relevant applied 
R&D must meet additional criteria. Together, these 
criteria can be used to assess the need, relevance, 
appropriateness, quality, and performance of federal R&D 
programs. 

I. Relevance 


R&D investments must have clear plans, must be 
relevant to national priorities, agency missions, 
relevant fields, and "customer" needs, and must 
justify their claim on taxpayer resources. Programs 
that directly support Presidential priorities may 
receive special consideration with adequate 
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documentation of their relevance. Review committees 
should assess program objectives and goals on their 
relevance to national needs, "customer" needs, agency 
missions, and the field(s) of study the program 
strives to address. For example, the Joint DOE/NSF 
Nuclear Sciences Advisory Committee's Long Range Plan 
and the Astronomy Decadal Surveys are the products of 
good planning processes because they articulate goals 
and priorities for research opportunities within and 
across their respective fields. 

0MB will work with some programs to identify 
quantitative metrics to estimate and compare potential 
benefits across programs with similar goals. Such 
comparisons may be within an agency or among agencies. 

A. Programs must have complete plans, with clear 
goals and priorities. 

Programs must provide complete plans, which include 
explicit statements of: 

- specific issues motivating the program; 

- broad goals and more specific tasks meant to 
address the issues; 

- priorities among goals and activities within 
the program; 

- human and capital resources anticipated; and 

- intended program outcomes, against which 
success may later be assessed. 

B. Programs must articulate the potential public 
benefits of the program. 

Programs must identify potential benefits, including 
added benefits beyond those of any similar efforts 
that have been or are being funded by the government 
or others. R&D benefits may include technologies and 
methods that could provide new options in the 
future, if the landscape of today's needs and 
capabilities changes dramatically. Some programs and 
sub-program units may be required to quantitatively 
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estimate expected benefits, which would include 
metrics to permit meaningful comparisons among 
programs that promise similar benefits. While all 
programs should try to articulate potential 
benefits, 0MB and OSTP recognize the difficulty in 
predicting the outcomes of basic research. 
Consequently, agencies may be allowed to relax this 
as a requirement for basic research programs. 


C. Programs must document their relevance to specific 
Presidential priorities to receive special 
consideration. 

Many areas of research warrant some level of federal 
funding. Nonetheless, the President has identified a 
few specific areas of research that are particularly 
important. To the extent a proposed project can 
document how it directly addresses one of these 
areas, it may be given preferential treatment. 


D. Program relevance to the needs of the Nation, of 
fields of Science & Technology, and of program 

"customers" must be assessed through prospective 
external review. 

Programs must be assessed on their relevance to 

agency missions, fields of science or technology, or 
other "customer" needs. A customer may be another 
program at the same or another agency, an 

interagency initiative or partnership, or a firm or 
other organization from another sector or country. 
As appropriate, programs must define a plan for 
regular reviews by primary customers of the 

program's relevance to their needs. These programs 
must provide a plan for addressing the conclusions 
of external reviews. 


E. Program relevance to the needs of the Nation, of 
fields of S&T, and of program "customers" must be 
assessed periodically through retrospective external 
review. 

Programs must periodically assess the need for the 
program and its relevance to customers against the 
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original justifications. Programs must provide a plan 
for addressing the conclusions of external reviews. 

II. Quality 

Programs should maximize the quality of the R&D they fund 
through the use of a clearly stated, defensible method 
for awarding a significant majority of their funding. A 
customary method for promoting R&D quality is the use of 
a competitive, merit-based process. NSF's process for the 
peer-reviewed, competitive award of its R&D grants is a 
good example. Justifications for processes other than 
competitive merit review may include "outside-the-box" 
thinking, a need for timeliness (e.g., R&D grants for 
rapid response studies of Pfisteria), unique skills or 
facilities, or a proven record of outstanding performance 
(e.g., performance-based renewals). 

Programs must assess and report on the quality of current 
and past R&D. For example, NSF's use of Committees of 
Visitors, which review NSF directorates, is an example of 
a good quality-assessment tool. 0MB and OSTP encourage 
agencies to provide the means by which their programs may 
be benchmarked internationally or across agencies, which 
provides one indicator of program quality. 


A. Programs allocating funds through means other than a 
competitive, merit-based process must justify funding 
methods and document how quality is maintained. 

Programs must clearly describe how much of the 
requested funding will be broadly competitive based on 
merit, providing compelling justifications for R&D 
funding allocated through other means. (See 0MB 
Circular A-11 for definitions of competitive merit 
review and other means of allocating federal research 
funding.) All program funds allocated through means 
other than unlimited competition must document the 
processes they will use to distribute funds to each 
type of R&D performer (e.g., federal laboratories, 
federally funded R&D centers, universities, etc.). 
Programs are encouraged to use external assessment of 
the methods they use to allocate R&D and maintain 
program quality. 
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B. Program quality must be assessed periodically through 
retrospective expert review. 

Programs must institute a plan for regular, external 
reviews of the quality of the program's research and 
research performers, including a plan to use the 
results from these reviews to guide future program 
decisions. Rolling reviews performed every 3-5 years 
by advisory committees can satisfy this requirement. 
Benchmarking of scientific leadership and other 
factors provides an effective means of assessing 
program quality relative to other programs, other 
agencies, and other countries. 

III. Performance 

R&D programs should maintain a set of high priority, 
multi-year R&D objectives with annual performance outputs 
and milestones that show how one or more outcomes will be 
reached. Metrics should be defined not only to encourage 
individual program performance but also to promote, as 
appropriate, broader goals, such as innovation, 
cooperation, education, and dissemination of knowledge, 
applications, or tools. 

0MB encourages agencies to make the processes they use to 
satisfy the Government Performance and Results Act (GRPA) 
consistent with the goals and metrics they use to satisfy 
these R&D criteria. Satisfying the R&D performance 
criteria for a given program should serve to set and 
evaluate R&D performance goals for the purposes of GPRA. 
0MB expects goals and performance measures that satisfy 
the R&D criteria to be reflected in agency performance 
plans. 

Programs must demonstrate an ability to manage in a 
manner that produces identifiable results. At the same 
time, taking risks and working toward difficult-to-attain 
goals are important aspects of good research management, 
especially for basic research. The intent of the 
investment criteria is not to drive basic research 
programs to pursue less risky research that has a greater 
chance of success. Instead, the Administration will focus 
on improving the management of basic research programs. 

0MB will work with some programs to identify quantitative 
metrics to compare performance across programs with 
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similar goals. Such comparisons may be within an agency 
or among agencies. 

Construction projects and facility operations will 
require additional performance metrics. Cost and schedule 
earned-value metrics for the construction of R&D 
facilities must be tracked and reported. Within DOE, the 
Office of Science's formalized independent reviews of 
technical cost, scope, and schedule baselines and project 
management of construction projects ("Lehman Reviews") 
are widely recognized as an effective practice for 
discovering and correcting problems involved with 
complex, one-of-a-kind construction projects. 


A. Programs may be required to track and report relevant 
program inputs annually. 

Programs may be expected to report relevant program 
inputs, which could include statistics on overhead, 
intramural/extramural spending, infrastructure, and 
human capital. These inputs should be discussed with 
0MB. 

B. Programs must define appropriate output and outcome 
measures, schedules, and decision points. 

Programs must provide single- and multi-year R&D 
objectives, with annual performance outputs, to track 
how the program will improve scientific understanding 
and its application. Programs must provide schedules 
with annual milestones for future competitions, 
decisions, and termination points, highlighting 
changes from previous schedules. Program proposals 
must define what would be a minimally effective 
program and a successful program. Agencies should 
define appropriate output and outcome measures for all 
R&D programs, but agencies should not expect 
fundamental basic research to be able to identify 
outcomes and measure performance in the same way that 
applied research or development are able to. 
Highlighting the results of basic research is 
important, but it should not come at the expense of 
risk-taking and innovation. For some basic research 
programs, 0MB may accept the use of qualitative 
outcome measures and quantitative process metrics. 
Facilities programs must define metrics and methods 
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(e.g., earned-value reporting) to track development 
costs and to assess the use and needs of operational 
facilities over time. If leadership in a particular 
field is a goal for a program or agency, 0MB and OSTP 
encourage the use of benchmarks to assess the 
processes and outcomes of the program with respect to 
leadership. 0MB encourages agencies to make the 
processes they use to satisfy GPRA consistent with the 
goals and metrics they use to satisfy these R&D 
criteria. 


C. Program performance must be retrospectively documented 
annually. 

Programs must document performance against previously 
defined output and outcome metrics, including progress 
toward objectives, decisions, and termination points 
or other transitions. Programs with similar goals may 
be compared on the basis of their performance. 0MB 
will work with agencies to identify such programs and 
appropriate metrics to enable such comparisons. 

IV. Criteria for R&D Programs Developing Technologies That 
Address Industry Issues 

The purpose of some R&D and technology demonstration 
programs and projects is to introduce some product or 
concept into the marketplace. However, some of these 
efforts engage in activities that industry is capable of 
doing and may discourage or even displace industry 
investment that would occur otherwise. For the purposes 
of assessing federal R&D investments, the following 
criteria should be used to assess industry-relevant R&D 
and demonstration projects, including, at 0MB discretion, 
associated construction activities. 

0MB will work with programs to identify quantitative 
metrics to measure and compare potential benefits and 
performance across programs with similar goals, as well 
as ways to assess market relevance. 


A. Programs and projects must articulate public benefits 
of the program using uniform benefit indicators across 
programs and projects with similar goals. 
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In addition to the public benefits required in the 
general criteria, all industry-relevant programs and 
projects must identify and use uniform benefit 
indicators (including benefit-cost ratios) to enable 
comparisons of expected benefits across programs and 
projects. 0MB will work with agencies to identify 
these indicators. 

B. Programs and projects must justify the appropriateness 

of federal investment, including the manner in which 
the market fails to motivate private sector 

investment. 

A lack of market incentives discourages private firms 
from investing in research where the benefits may 
occur far in the future, the risks may be too great 
for non-federal participants, or the benefits accrue 
to the public rather than private investors. Programs 
and projects must demonstrate that industry investment 
is sub-optimal and explain in what way the market 
fails that prevents the private sector from capturing 
the benefits of developing the good or service. 

C. Programs and projects must demonstrate that investment 
in R&D and demonstration activities is the best means 
to support the federal policy goals, compared to other 
policy alternatives. 

When the federal government chooses to intervene to 
address market failures, there may be many policy 
alternatives to address those failures. Among the 
other tools available to the government are 
legislation, tax policy, regulatory and enforcement 
efforts, and an integrated combination of these 
approaches. In this context, projects to address 
issues of genuine federal concern should be able to 
illustrate how R&D and demonstration activities are 
superior to other policy tools in addressing federal 
goals, either by themselves or as part of an 
integrated package. 


D. Programs and projects must document industry or market 
relevance, including readiness of the market to adopt 
technologies or other outputs. 
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Programs must assess the likelihood that the target 
industry will be able to adopt the technology or other 
program outputs. The level of industry cost sharing is 
one indicator of industry relevance. Before projects 
move into demonstration or deployment stages, an 
economic analysis of the public and private returns on 
the public investment must be provided. 


E. Program performance plans and reports must include 
"off ramps" and transition points. 

In addition to the schedules and decision points 
defined in the general criteria, program plans should 
also identify whether, when, and how aspects of the 
program may be shifted to the private sector. 
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APPENDIX C. SCORECARD CORE CRITERIA 


(Source: http://www.john-mercer.com/pma-stds5.htm) 


Criteria for "Green" 

In order to be recognized as achieving full compliance 

with the Budget and Performance Integration initiative of 

the President's Management Agenda, the agency must satisfy 

0MB that it meets each of the following six Core Criteria: 

o "Senior agency managers meet at least quarterly 
to examine reports that integrate financial and 
performance information that covers all major 
responsibilities of the Department. This 
information is used to make decisions regarding 
the management of agency programs;" 
o "Strategic plans contain a limited number of 
outcome-oriented goals and objectives. Annual 
budget and performance documents incorporate 
all measures identified in the PART and focus 
on the information used in the senior 

management report described in the first 
criterion;" 

o "Performance appraisal plans for at least 60% 
of agency positions link to agency mission, 
goals and outcomes, effectively differentiate 
between various levels of performance, and 
provide consequences based on performance;" 
o "Reports the full cost of achieving performance 
goals accurately (+/-10%) in budget and 

performance documents and can accurately 
estimate the marginal cost (+/-10%) of changing 
performance goals;" "Has at least one 
efficiency measure for all programs; and" 
o "Uses PART evaluations to direct program 
improvements and PART ratings are used 
consistently to justify funding requests, 
management actions, and legislative proposals. 

Less than 10% of agency programs receive a 
Results Not Demonstrated rating for more than 
two years in a row." 
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Criteria for "Yellow" 


A rating of "yellow" is awarded when the agency meets 
the following criteria: 

o "Senior agency managers meet at least quarterly 
to examine reports that integrate financial and 
performance information that covers some of the 
major responsibilities of the Department. The 
reports are used to make decisions regarding 
the management of Agency programs;" 
o "Strategic plans contain a limited number of 
outcome-oriented goals and objectives. Annual 
budget and performance documents incorporate 
all measures identified in the PART process;" 
o "Performance appraisal plans for SES and 
managers link to agency mission, goals and 
outcomes, effectively differentiate between 
various levels of performance, and provide 
consequences based on performance;" 
o "The full cost of achieving performance goals 
is accurately (+/-10%) reported in budget and 
performance documents;" 

o "At least 50% of agency programs rated by the 
PART have at least one efficiency measure;" 
o "PART ratings are used to justify funding 
requests, management actions, and legislative 
proposals. No more than 50% of agency programs 
receive a Results Not Demonstrated rating for 
more than two years in a row." 


Negative Conditions for "Red" 

A rating of "red" is triggered if any one of the 
following negative Conditions is found: 

o "Senior agency managers do not have a regular 
process for considering financial and 
performance information when making decisions 
regarding the management of Agency programs;" 

o "Strategic plans contain too many goals and 
objectives to provide a clear and focused 
statement of Agency priorities. Performance 
measures included in annual budget and 
performance documents do not meet the standards 
of the PART;" 
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o 


"Performance appraisal plans do not link to 
agency mission, goals and outcomes, effectively 
differentiate between various levels of 
performance, or provide consequences based on 
performance;" 
o "Does not have a systematic way to estimate the 
full cost of achieving performance goals 
reported in budget and performance documents;" 
o "Less than 50% of agency programs rated by the 
PART have at least one efficiency measure;" OR 
o "Agency does not consistently use PART ratings 
to justify funding requests, management 
actions, and legislative proposals. More than 
50% of agency programs receive a Results Not 
Demonstrated rating for more than two years in 
a row." 
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